pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1597771089977544704 Results

Error lines from build-log.txt

Raw build-log.txt
... skipping 236 lines ...

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11345  100 11345    0     0   175k      0 --:--:-- --:--:-- --:--:--  178k
Downloading https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb || make container-all push-manifest
Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb not found: manifest unknown: manifest tagged by "latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=ae2d55c668f6568106b7dd40ab11a7c3dbac74fb -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2022-11-30T02:01:30Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin
docker buildx rm container-builder || true
ERROR: no builder "container-builder" found
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
# https://github.com/docker/buildx/issues/464#issuecomment-741507760
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64
Unable to find image 'tonistiigi/binfmt:latest' locally
... skipping 702 lines ...
         }
      }
   ]
}
make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender
Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb not found: manifest unknown: manifest tagged by "latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx rm container-builder || true
container-builder removed
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
... skipping 1031 lines ...
                    type: string
                type: object
                oneOf:
                - required: ["persistentVolumeClaimName"]
                - required: ["volumeSnapshotContentName"]
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 60 lines ...
                    type: string
                  volumeSnapshotContentName:
                    description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable.
                    type: string
                type: object
              volumeSnapshotClassName:
                description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.'
                type: string
            required:
            - source
            type: object
          status:
            description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.
... skipping 2 lines ...
                description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.'
                type: string
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown.
                format: date-time
                type: string
              error:
                description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                type: string
                description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                x-kubernetes-int-or-string: true
            type: object
        required:
        - spec
        type: object
... skipping 254 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 108 lines ...
            description: status represents the current information of a snapshot.
            properties:
              creationTime:
                description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC.
                format: int64
                type: integer
              error:
                description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared.
                properties:
                  message:
                    description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.'
                    type: string
                  time:
                    description: time is the timestamp when the error was encountered.
                    format: date-time
                    type: string
                type: object
              readyToUse:
                description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown.
                type: boolean
              restoreSize:
                description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown.
                format: int64
                minimum: 0
                type: integer
              snapshotHandle:
                description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress.
                type: string
            type: object
        required:
        - spec
        type: object
    served: true
... skipping 359 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 1063 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 385 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:242[0m

    [36mDistro debian doesn't support ntfs -- skipping[0m

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
... skipping 259 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents","total":36,"completed":1,"skipped":89,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:15:07.777: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 212 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:138[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":44,"completed":1,"skipped":11,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
... skipping 51 lines ...
Nov 30 02:12:40.101: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 30 02:12:40.250: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comrntfd] to have phase Bound
Nov 30 02:12:40.367: INFO: PersistentVolumeClaim test.csi.azure.comrntfd found but phase is Pending instead of Bound.
Nov 30 02:12:42.478: INFO: PersistentVolumeClaim test.csi.azure.comrntfd found but phase is Pending instead of Bound.
Nov 30 02:12:44.589: INFO: PersistentVolumeClaim test.csi.azure.comrntfd found and phase=Bound (4.33902468s)
[1mSTEP[0m: Creating pod to format volume volume-prep-provisioning-7056
Nov 30 02:12:44.979: INFO: Waiting up to 5m0s for pod "volume-prep-provisioning-7056" in namespace "provisioning-7056" to be "Succeeded or Failed"
Nov 30 02:12:45.098: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 119.434056ms
Nov 30 02:12:47.210: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 2.231282075s
Nov 30 02:12:49.320: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 4.341713063s
Nov 30 02:12:51.431: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 6.452555108s
Nov 30 02:12:53.542: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 8.563305155s
Nov 30 02:12:55.653: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 10.674837247s
... skipping 12 lines ...
Nov 30 02:13:23.100: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 38.121576272s
Nov 30 02:13:25.212: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 40.233109428s
Nov 30 02:13:32.603: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 47.624682375s
Nov 30 02:13:34.716: INFO: Pod "volume-prep-provisioning-7056": Phase="Pending", Reason="", readiness=false. Elapsed: 49.737007225s
Nov 30 02:13:36.827: INFO: Pod "volume-prep-provisioning-7056": Phase="Succeeded", Reason="", readiness=false. Elapsed: 51.84818608s
[1mSTEP[0m: Saw pod success
Nov 30 02:13:36.827: INFO: Pod "volume-prep-provisioning-7056" satisfied condition "Succeeded or Failed"
Nov 30 02:13:36.827: INFO: Deleting pod "volume-prep-provisioning-7056" in namespace "provisioning-7056"
Nov 30 02:13:36.952: INFO: Wait up to 5m0s for pod "volume-prep-provisioning-7056" to be fully deleted
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-27w5
[1mSTEP[0m: Checking for subpath error in container status
Nov 30 02:13:59.411: INFO: Deleting pod "pod-subpath-test-dynamicpv-27w5" in namespace "provisioning-7056"
Nov 30 02:13:59.523: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-27w5" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 30 02:14:01.752: INFO: Deleting pod "pod-subpath-test-dynamicpv-27w5" in namespace "provisioning-7056"
[1mSTEP[0m: Deleting pvc
Nov 30 02:14:01.862: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comrntfd"
... skipping 25 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should verify container cannot write to subpath readonly volumes [Slow]
    [90mtest/e2e/storage/testsuites/subpath.go:425[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should verify container cannot write to subpath readonly volumes [Slow]","total":34,"completed":1,"skipped":226,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumeIO
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:15:14.604: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 327 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":27,"completed":1,"skipped":28,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:15:49.218: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:280[0m

    [36mDistro debian doesn't support ntfs -- skipping[0m

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
... skipping 171 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:298[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":28,"completed":1,"skipped":48,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:16:35.419: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Inline-volume (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:242[0m

    [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 39 lines ...

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:280[0m

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 30 02:15:17.543: INFO: >>> kubeConfig: /root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename provisioning
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:280
Nov 30 02:15:18.318: INFO: Creating resource for dynamic PV
Nov 30 02:15:18.318: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass provisioning-5508-e2e-scv8qd2
[1mSTEP[0m: creating a claim
Nov 30 02:15:18.429: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 30 02:15:18.541: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comtx7tl] to have phase Bound
Nov 30 02:15:18.651: INFO: PersistentVolumeClaim test.csi.azure.comtx7tl found but phase is Pending instead of Bound.
Nov 30 02:15:20.762: INFO: PersistentVolumeClaim test.csi.azure.comtx7tl found but phase is Pending instead of Bound.
Nov 30 02:15:22.874: INFO: PersistentVolumeClaim test.csi.azure.comtx7tl found and phase=Bound (4.332578937s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-9ctz
[1mSTEP[0m: Checking for subpath error in container status
Nov 30 02:15:45.434: INFO: Deleting pod "pod-subpath-test-dynamicpv-9ctz" in namespace "provisioning-5508"
Nov 30 02:15:45.547: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-9ctz" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 30 02:15:47.772: INFO: Deleting pod "pod-subpath-test-dynamicpv-9ctz" in namespace "provisioning-5508"
[1mSTEP[0m: Deleting pvc
Nov 30 02:15:47.882: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comtx7tl"
... skipping 22 lines ...

[32m• [SLOW TEST:102.771 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:280[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]","total":34,"completed":2,"skipped":534,"failed":0}

[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:17:00.318: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping
... skipping 287 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with pvc data source
    [90mtest/e2e/storage/testsuites/provisioning.go:421[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source","total":31,"completed":1,"skipped":27,"failed":0}
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:17:18.336: INFO: Distro debian doesn't support ntfs -- skipping
[AfterEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/framework/framework.go:188

... skipping 42 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:280[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 193 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:138[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":44,"completed":2,"skipped":35,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:17:53.876: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 81 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Inline-volume (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:258[0m

    [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 69 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support two pods which have the same volume definition
    [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition","total":30,"completed":1,"skipped":4,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:17:59.949: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 140 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents","total":27,"completed":2,"skipped":97,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] provisioning
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] provisioning
... skipping 133 lines ...
Nov 30 02:17:55.896: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comtzzqn] to have phase Bound
Nov 30 02:17:56.004: INFO: PersistentVolumeClaim test.csi.azure.comtzzqn found but phase is Pending instead of Bound.
Nov 30 02:17:58.114: INFO: PersistentVolumeClaim test.csi.azure.comtzzqn found but phase is Pending instead of Bound.
Nov 30 02:18:00.224: INFO: PersistentVolumeClaim test.csi.azure.comtzzqn found and phase=Bound (4.327519916s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-qcb6
[1mSTEP[0m: Creating a pod to test atomic-volume-subpath
Nov 30 02:18:00.566: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-qcb6" in namespace "provisioning-5975" to be "Succeeded or Failed"
Nov 30 02:18:00.676: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Pending", Reason="", readiness=false. Elapsed: 109.962373ms
Nov 30 02:18:02.786: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219820111s
Nov 30 02:18:04.895: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329049484s
Nov 30 02:18:07.006: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43930734s
Nov 30 02:18:09.123: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.556316727s
Nov 30 02:18:11.233: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.666674577s
... skipping 18 lines ...
Nov 30 02:18:51.345: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Running", Reason="", readiness=true. Elapsed: 50.778624964s
Nov 30 02:18:53.457: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Running", Reason="", readiness=true. Elapsed: 52.89074829s
Nov 30 02:18:55.568: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Running", Reason="", readiness=true. Elapsed: 55.001702063s
Nov 30 02:18:57.677: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Running", Reason="", readiness=false. Elapsed: 57.110950617s
Nov 30 02:18:59.787: INFO: Pod "pod-subpath-test-dynamicpv-qcb6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 59.220961469s
[1mSTEP[0m: Saw pod success
Nov 30 02:18:59.787: INFO: Pod "pod-subpath-test-dynamicpv-qcb6" satisfied condition "Succeeded or Failed"
Nov 30 02:18:59.896: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000002 pod pod-subpath-test-dynamicpv-qcb6 container test-container-subpath-dynamicpv-qcb6: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:19:00.153: INFO: Waiting for pod pod-subpath-test-dynamicpv-qcb6 to disappear
Nov 30 02:19:00.283: INFO: Pod pod-subpath-test-dynamicpv-qcb6 no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-qcb6
Nov 30 02:19:00.283: INFO: Deleting pod "pod-subpath-test-dynamicpv-qcb6" in namespace "provisioning-5975"
... skipping 23 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support file as subpath [LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:232[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly]","total":44,"completed":3,"skipped":104,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:19:42.189: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 288 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:248[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":28,"completed":2,"skipped":235,"failed":0}

[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m 
  [1mshould concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS][0m
  [37mtest/e2e/storage/testsuites/multivolume.go:323[0m
... skipping 108 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:323[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":36,"completed":2,"skipped":121,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:19:57.097: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 118 lines ...
Nov 30 02:19:46.190: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.commngr9] to have phase Bound
Nov 30 02:19:46.298: INFO: PersistentVolumeClaim test.csi.azure.commngr9 found but phase is Pending instead of Bound.
Nov 30 02:19:48.409: INFO: PersistentVolumeClaim test.csi.azure.commngr9 found but phase is Pending instead of Bound.
Nov 30 02:19:50.518: INFO: PersistentVolumeClaim test.csi.azure.commngr9 found and phase=Bound (4.327415214s)
[1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-w8b6
[1mSTEP[0m: Creating a pod to test exec-volume-test
Nov 30 02:19:50.849: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-w8b6" in namespace "volume-5459" to be "Succeeded or Failed"
Nov 30 02:19:50.973: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 123.820837ms
Nov 30 02:19:53.084: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.233972228s
Nov 30 02:19:55.193: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.343960773s
Nov 30 02:19:57.309: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.459768211s
Nov 30 02:19:59.418: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.568787057s
Nov 30 02:20:01.528: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.67818491s
Nov 30 02:20:03.639: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 12.789021852s
Nov 30 02:20:05.750: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 14.900497187s
Nov 30 02:20:07.860: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 17.010961149s
Nov 30 02:20:09.970: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Pending", Reason="", readiness=false. Elapsed: 19.120770953s
Nov 30 02:20:12.083: INFO: Pod "exec-volume-test-dynamicpv-w8b6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 21.23363461s
[1mSTEP[0m: Saw pod success
Nov 30 02:20:12.083: INFO: Pod "exec-volume-test-dynamicpv-w8b6" satisfied condition "Succeeded or Failed"
Nov 30 02:20:12.193: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000002 pod exec-volume-test-dynamicpv-w8b6 container exec-container-dynamicpv-w8b6: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:20:12.424: INFO: Waiting for pod exec-volume-test-dynamicpv-w8b6 to disappear
Nov 30 02:20:12.531: INFO: Pod exec-volume-test-dynamicpv-w8b6 no longer exists
[1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-w8b6
Nov 30 02:20:12.532: INFO: Deleting pod "exec-volume-test-dynamicpv-w8b6" in namespace "volume-5459"
... skipping 21 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should allow exec of files on the volume
    [90mtest/e2e/storage/testsuites/volumes.go:198[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume","total":28,"completed":3,"skipped":236,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:20:54.392: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 172 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:378[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":31,"completed":2,"skipped":138,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow][0m 
  [1mshould access to two volumes with the same volume mode and retain data across pod recreation on different node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:168[0m
... skipping 194 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:168[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":34,"completed":3,"skipped":746,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:22:21.886: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 115 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read/write inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume","total":27,"completed":3,"skipped":329,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m 
  [1mshould access to two volumes with different volume mode and retain data across pod recreation on the same node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:209[0m
... skipping 195 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:209[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":44,"completed":4,"skipped":165,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:22:39.166: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/volumemode.go:299[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 226 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:209[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":36,"completed":3,"skipped":215,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:23:13.634: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping
... skipping 147 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:323[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":30,"completed":2,"skipped":12,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:24:21.395: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 99 lines ...
Nov 30 02:22:40.326: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comckcrx] to have phase Bound
Nov 30 02:22:40.435: INFO: PersistentVolumeClaim test.csi.azure.comckcrx found but phase is Pending instead of Bound.
Nov 30 02:22:42.546: INFO: PersistentVolumeClaim test.csi.azure.comckcrx found but phase is Pending instead of Bound.
Nov 30 02:22:44.656: INFO: PersistentVolumeClaim test.csi.azure.comckcrx found and phase=Bound (4.329722742s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-4nzt
[1mSTEP[0m: Creating a pod to test subpath
Nov 30 02:22:44.987: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-4nzt" in namespace "provisioning-2557" to be "Succeeded or Failed"
Nov 30 02:22:45.100: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 112.80226ms
Nov 30 02:22:47.209: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.2220527s
Nov 30 02:22:49.318: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.331677007s
Nov 30 02:22:51.434: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.44723366s
Nov 30 02:22:53.546: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.559224835s
Nov 30 02:22:55.667: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.680743754s
... skipping 35 lines ...
Nov 30 02:24:11.673: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.686044115s
Nov 30 02:24:13.789: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.802550886s
Nov 30 02:24:15.900: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.913239065s
Nov 30 02:24:18.022: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m33.034954911s
Nov 30 02:24:20.131: INFO: Pod "pod-subpath-test-dynamicpv-4nzt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m35.144266322s
[1mSTEP[0m: Saw pod success
Nov 30 02:24:20.131: INFO: Pod "pod-subpath-test-dynamicpv-4nzt" satisfied condition "Succeeded or Failed"
Nov 30 02:24:20.240: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000002 pod pod-subpath-test-dynamicpv-4nzt container test-container-subpath-dynamicpv-4nzt: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:24:20.495: INFO: Waiting for pod pod-subpath-test-dynamicpv-4nzt to disappear
Nov 30 02:24:20.604: INFO: Pod pod-subpath-test-dynamicpv-4nzt no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-4nzt
Nov 30 02:24:20.604: INFO: Deleting pod "pod-subpath-test-dynamicpv-4nzt" in namespace "provisioning-2557"
... skipping 29 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support existing single file [LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:221[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","total":44,"completed":5,"skipped":307,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] provisioning[0m 
  [1mshould provision storage with pvc data source in parallel [Slow][0m
  [37mtest/e2e/storage/testsuites/provisioning.go:459[0m
... skipping 372 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with pvc data source in parallel [Slow]
    [90mtest/e2e/storage/testsuites/provisioning.go:459[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source in parallel [Slow]","total":28,"completed":4,"skipped":361,"failed":0}

[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:25:59.358: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Inline-volume (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:269[0m

    [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 201 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:138[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":36,"completed":4,"skipped":325,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource][0m [0mvolume snapshot controller[0m [90m[0m 
  [1mshould check snapshot fields, check restore correctly works, check deletion (ephemeral)[0m
  [37mtest/e2e/storage/testsuites/snapshottable.go:177[0m
... skipping 10 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 30 02:22:22.852: INFO: Creating resource for dynamic PV
Nov 30 02:22:22.853: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass snapshotting-1-e2e-scqshcp
[1mSTEP[0m: [init] starting a pod to use the claim
Nov 30 02:22:23.079: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-lpfrx" in namespace "snapshotting-1" to be "Succeeded or Failed"
Nov 30 02:22:23.190: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 110.358933ms
Nov 30 02:22:25.301: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221472986s
Nov 30 02:22:27.429: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.349719371s
Nov 30 02:22:29.540: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.460455663s
Nov 30 02:22:31.650: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.570839654s
Nov 30 02:22:33.761: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.682046567s
... skipping 24 lines ...
Nov 30 02:23:26.573: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.493292651s
Nov 30 02:23:28.685: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.605409817s
Nov 30 02:23:30.796: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.716838601s
Nov 30 02:23:32.908: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.82882345s
Nov 30 02:23:35.022: INFO: Pod "pvc-snapshottable-tester-lpfrx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m11.943106781s
[1mSTEP[0m: Saw pod success
Nov 30 02:23:35.023: INFO: Pod "pvc-snapshottable-tester-lpfrx" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
[1mSTEP[0m: creating a SnapshotClass
[1mSTEP[0m: creating a dynamic VolumeSnapshot
Nov 30 02:23:35.486: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-cs6tq to become ready
Nov 30 02:23:35.597: INFO: VolumeSnapshot snapshot-cs6tq found but is not ready.
Nov 30 02:23:37.708: INFO: VolumeSnapshot snapshot-cs6tq found but is not ready.
... skipping 39 lines ...
Nov 30 02:25:36.689: INFO: volumesnapshotcontents snapcontent-ff07ffde-4a78-4fb9-98f5-c6db86583be1 has been found and is not deleted
Nov 30 02:25:37.799: INFO: volumesnapshotcontents snapcontent-ff07ffde-4a78-4fb9-98f5-c6db86583be1 has been found and is not deleted
Nov 30 02:25:38.912: INFO: volumesnapshotcontents snapcontent-ff07ffde-4a78-4fb9-98f5-c6db86583be1 has been found and is not deleted
Nov 30 02:25:40.022: INFO: volumesnapshotcontents snapcontent-ff07ffde-4a78-4fb9-98f5-c6db86583be1 has been found and is not deleted
Nov 30 02:25:41.133: INFO: volumesnapshotcontents snapcontent-ff07ffde-4a78-4fb9-98f5-c6db86583be1 has been found and is not deleted
Nov 30 02:25:42.244: INFO: volumesnapshotcontents snapcontent-ff07ffde-4a78-4fb9-98f5-c6db86583be1 has been found and is not deleted
Nov 30 02:25:43.244: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 30 02:25:43.388: INFO: Pod restored-pvc-tester-zv8rl has the following logs: 
Nov 30 02:25:43.388: INFO: Deleting pod "restored-pvc-tester-zv8rl" in namespace "snapshotting-1"
Nov 30 02:25:43.501: INFO: Wait up to 5m0s for pod "restored-pvc-tester-zv8rl" to be fully deleted
Nov 30 02:26:15.721: INFO: deleting snapshot "snapshotting-1"/"snapshot-cs6tq"
... skipping 26 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        [90mtest/e2e/storage/testsuites/snapshottable.go:177[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":34,"completed":4,"skipped":887,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (ntfs)][Feature:Windows] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:26:23.692: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 59 lines ...
Nov 30 02:22:25.729: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comx5z4t] to have phase Bound
Nov 30 02:22:25.838: INFO: PersistentVolumeClaim test.csi.azure.comx5z4t found but phase is Pending instead of Bound.
Nov 30 02:22:27.950: INFO: PersistentVolumeClaim test.csi.azure.comx5z4t found but phase is Pending instead of Bound.
Nov 30 02:22:30.060: INFO: PersistentVolumeClaim test.csi.azure.comx5z4t found and phase=Bound (4.33061982s)
[1mSTEP[0m: [init] starting a pod to use the claim
[1mSTEP[0m: [init] check pod success
Nov 30 02:22:30.498: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-jmn98" in namespace "snapshotting-1971" to be "Succeeded or Failed"
Nov 30 02:22:30.607: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 108.281604ms
Nov 30 02:22:32.716: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217890701s
Nov 30 02:22:34.827: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328902798s
Nov 30 02:22:36.938: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439473451s
Nov 30 02:22:39.047: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549082332s
Nov 30 02:22:41.158: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 10.659918092s
... skipping 43 lines ...
Nov 30 02:24:14.049: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 1m43.550353319s
Nov 30 02:24:16.160: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 1m45.661233691s
Nov 30 02:24:18.270: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 1m47.771718647s
Nov 30 02:24:20.380: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Pending", Reason="", readiness=false. Elapsed: 1m49.882062649s
Nov 30 02:24:22.492: INFO: Pod "pvc-snapshottable-tester-jmn98": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m51.99358958s
[1mSTEP[0m: Saw pod success
Nov 30 02:24:22.492: INFO: Pod "pvc-snapshottable-tester-jmn98" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
Nov 30 02:24:22.604: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comx5z4t] to have phase Bound
Nov 30 02:24:22.727: INFO: PersistentVolumeClaim test.csi.azure.comx5z4t found and phase=Bound (123.433363ms)
[1mSTEP[0m: [init] checking the PV
[1mSTEP[0m: [init] deleting the pod
Nov 30 02:24:23.086: INFO: Pod pvc-snapshottable-tester-jmn98 has the following logs: 
... skipping 12 lines ...
Nov 30 02:24:28.313: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-74b15422-9db2-4223-8224-d06340ff44bf creationTime:2022-11-30T02:24:24Z readyToUse:true restoreSize:5Gi]
Nov 30 02:24:28.313: INFO: snapshotContentName snapcontent-74b15422-9db2-4223-8224-d06340ff44bf
[1mSTEP[0m: checking the snapshot
[1mSTEP[0m: checking the SnapshotContent
[1mSTEP[0m: Modifying source data test
[1mSTEP[0m: modifying the data in the source PVC
Nov 30 02:24:28.751: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-97hlp" in namespace "snapshotting-1971" to be "Succeeded or Failed"
Nov 30 02:24:28.859: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 108.241253ms
Nov 30 02:24:30.969: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217958191s
Nov 30 02:24:33.079: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.32855823s
Nov 30 02:24:35.190: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438913827s
Nov 30 02:24:37.299: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.548381061s
Nov 30 02:24:39.409: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.658051524s
... skipping 24 lines ...
Nov 30 02:25:32.152: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.401824683s
Nov 30 02:25:34.262: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.511551293s
Nov 30 02:25:36.372: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.621466991s
Nov 30 02:25:38.481: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.730732576s
Nov 30 02:25:40.591: INFO: Pod "pvc-snapshottable-data-tester-97hlp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m11.840047791s
[1mSTEP[0m: Saw pod success
Nov 30 02:25:40.591: INFO: Pod "pvc-snapshottable-data-tester-97hlp" satisfied condition "Succeeded or Failed"
Nov 30 02:25:40.812: INFO: Pod pvc-snapshottable-data-tester-97hlp has the following logs: 
Nov 30 02:25:40.812: INFO: Deleting pod "pvc-snapshottable-data-tester-97hlp" in namespace "snapshotting-1971"
Nov 30 02:25:40.926: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-97hlp" to be fully deleted
[1mSTEP[0m: creating a pvc from the snapshot
[1mSTEP[0m: starting a pod to use the snapshot
Nov 30 02:26:21.532: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-5bbt3k3k.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-1971 exec restored-pvc-tester-pdc7v --namespace=snapshotting-1971 -- cat /mnt/test/data'
... skipping 47 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":27,"completed":4,"skipped":341,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] volumeLimits[0m 
  [1mshould verify that all csinodes have volume limits[0m
  [37mtest/e2e/storage/testsuites/volumelimits.go:249[0m
... skipping 16 lines ...
  test/e2e/framework/framework.go:188
Nov 30 02:27:09.132: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[1mSTEP[0m: Destroying namespace "volumelimits-3122" for this suite.

[32m•[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits should verify that all csinodes have volume limits","total":27,"completed":5,"skipped":468,"failed":0}

[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:27:09.356: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 125 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should store data
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data","total":31,"completed":3,"skipped":152,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral[0m 
  [1mshould support multiple inline ephemeral volumes[0m
  [37mtest/e2e/storage/testsuites/ephemeral.go:254[0m
... skipping 51 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support multiple inline ephemeral volumes
    [90mtest/e2e/storage/testsuites/ephemeral.go:254[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":44,"completed":6,"skipped":319,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould support restarting containers using file as subpath [Slow][LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:333[0m
... skipping 65 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support restarting containers using file as subpath [Slow][LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:333[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using file as subpath [Slow][LinuxOnly]","total":30,"completed":3,"skipped":136,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
... skipping 90 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read/write inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":28,"completed":5,"skipped":390,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:28:57.877: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 34 lines ...

    test/e2e/storage/testsuites/volume_expand.go:95
[90m------------------------------[0m
[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:269[0m

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 30 02:27:40.356: INFO: >>> kubeConfig: /root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename provisioning
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:269
Nov 30 02:27:41.117: INFO: Creating resource for dynamic PV
Nov 30 02:27:41.117: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass provisioning-9223-e2e-scd7trw
[1mSTEP[0m: creating a claim
Nov 30 02:27:41.234: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 30 02:27:41.348: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com5drsx] to have phase Bound
Nov 30 02:27:41.457: INFO: PersistentVolumeClaim test.csi.azure.com5drsx found but phase is Pending instead of Bound.
Nov 30 02:27:43.567: INFO: PersistentVolumeClaim test.csi.azure.com5drsx found but phase is Pending instead of Bound.
Nov 30 02:27:45.678: INFO: PersistentVolumeClaim test.csi.azure.com5drsx found and phase=Bound (4.329034992s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-dqpb
[1mSTEP[0m: Checking for subpath error in container status
Nov 30 02:28:34.230: INFO: Deleting pod "pod-subpath-test-dynamicpv-dqpb" in namespace "provisioning-9223"
Nov 30 02:28:34.341: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-dqpb" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 30 02:28:36.559: INFO: Deleting pod "pod-subpath-test-dynamicpv-dqpb" in namespace "provisioning-9223"
[1mSTEP[0m: Deleting pvc
Nov 30 02:28:36.667: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com5drsx"
... skipping 22 lines ...

[32m• [SLOW TEST:128.631 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:269[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]","total":44,"completed":7,"skipped":338,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (xfs)][Slow] volumes[0m 
  [1mshould allow exec of files on the volume[0m
  [37mtest/e2e/storage/testsuites/volumes.go:198[0m
... skipping 17 lines ...
Nov 30 02:26:24.710: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comrdjd2] to have phase Bound
Nov 30 02:26:24.821: INFO: PersistentVolumeClaim test.csi.azure.comrdjd2 found but phase is Pending instead of Bound.
Nov 30 02:26:26.931: INFO: PersistentVolumeClaim test.csi.azure.comrdjd2 found but phase is Pending instead of Bound.
Nov 30 02:26:29.041: INFO: PersistentVolumeClaim test.csi.azure.comrdjd2 found and phase=Bound (4.331391302s)
[1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-4r9v
[1mSTEP[0m: Creating a pod to test exec-volume-test
Nov 30 02:26:29.370: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-4r9v" in namespace "volume-8102" to be "Succeeded or Failed"
Nov 30 02:26:29.480: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 109.095177ms
Nov 30 02:26:31.604: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 2.233937627s
Nov 30 02:26:33.716: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 4.345079202s
Nov 30 02:26:35.826: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 6.455299969s
Nov 30 02:26:37.936: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 8.565454545s
Nov 30 02:26:40.046: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 10.675855491s
... skipping 24 lines ...
Nov 30 02:27:32.820: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.449196575s
Nov 30 02:27:34.930: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.559356368s
Nov 30 02:27:37.040: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.669767118s
Nov 30 02:27:39.150: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.779928635s
Nov 30 02:27:41.260: INFO: Pod "exec-volume-test-dynamicpv-4r9v": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m11.889680692s
[1mSTEP[0m: Saw pod success
Nov 30 02:27:41.260: INFO: Pod "exec-volume-test-dynamicpv-4r9v" satisfied condition "Succeeded or Failed"
Nov 30 02:27:41.382: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000001 pod exec-volume-test-dynamicpv-4r9v container exec-container-dynamicpv-4r9v: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:27:41.639: INFO: Waiting for pod exec-volume-test-dynamicpv-4r9v to disappear
Nov 30 02:27:41.749: INFO: Pod exec-volume-test-dynamicpv-4r9v no longer exists
[1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-4r9v
Nov 30 02:27:41.749: INFO: Deleting pod "exec-volume-test-dynamicpv-4r9v" in namespace "volume-8102"
... skipping 39 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should allow exec of files on the volume
    [90mtest/e2e/storage/testsuites/volumes.go:198[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should allow exec of files on the volume","total":34,"completed":5,"skipped":973,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:29:55.562: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 13 lines ...

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould fail if subpath file is outside the volume [Slow][LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:258[0m

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 30 02:28:57.885: INFO: >>> kubeConfig: /root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename provisioning
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath file is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:258
Nov 30 02:28:58.651: INFO: Creating resource for dynamic PV
Nov 30 02:28:58.651: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass provisioning-6217-e2e-sc265tw
[1mSTEP[0m: creating a claim
Nov 30 02:28:58.762: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 30 02:28:58.873: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comhnt2n] to have phase Bound
Nov 30 02:28:58.982: INFO: PersistentVolumeClaim test.csi.azure.comhnt2n found but phase is Pending instead of Bound.
Nov 30 02:29:01.092: INFO: PersistentVolumeClaim test.csi.azure.comhnt2n found but phase is Pending instead of Bound.
Nov 30 02:29:03.201: INFO: PersistentVolumeClaim test.csi.azure.comhnt2n found but phase is Pending instead of Bound.
Nov 30 02:29:05.311: INFO: PersistentVolumeClaim test.csi.azure.comhnt2n found and phase=Bound (6.437929739s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-57sp
[1mSTEP[0m: Checking for subpath error in container status
Nov 30 02:29:43.863: INFO: Deleting pod "pod-subpath-test-dynamicpv-57sp" in namespace "provisioning-6217"
Nov 30 02:29:43.974: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-57sp" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 30 02:29:46.194: INFO: Deleting pod "pod-subpath-test-dynamicpv-57sp" in namespace "provisioning-6217"
[1mSTEP[0m: Deleting pvc
Nov 30 02:29:46.303: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comhnt2n"
... skipping 16 lines ...

[32m• [SLOW TEST:90.069 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail if subpath file is outside the volume [Slow][LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:258[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow][LinuxOnly]","total":28,"completed":6,"skipped":407,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:30:28.018: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 37 lines ...
  test/e2e/framework/framework.go:188
Nov 30 02:30:29.269: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[1mSTEP[0m: Destroying namespace "volumelimits-1465" for this suite.

[32m•[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits should verify that all csinodes have volume limits","total":28,"completed":7,"skipped":501,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m 
  [1mshould concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS][0m
  [37mtest/e2e/storage/testsuites/multivolume.go:378[0m
... skipping 86 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:378[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":31,"completed":4,"skipped":221,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (xfs)][Slow] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:30:53.093: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 147 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should store data
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should store data","total":30,"completed":4,"skipped":312,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:32:50.984: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 206 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support two pods which have the same volume definition
    [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition","total":36,"completed":5,"skipped":360,"failed":0}

[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:33:11.642: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping
... skipping 102 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should not mount / map unused volumes in a pod [LinuxOnly]
    [90mtest/e2e/storage/testsuites/volumemode.go:354[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":31,"completed":5,"skipped":304,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:33:26.169: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Inline-volume (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:280[0m

    [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 26 lines ...
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 30 02:33:26.228: INFO: >>> kubeConfig: /root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename topology
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 30 02:33:26.984: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/framework/framework.go:188
Nov 30 02:33:26.985: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[1mSTEP[0m: Destroying namespace "topology-6982" for this suite.


[36m[1mS [SKIPPING] [1.090 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (delayed binding)] topology
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement][0m
    [90mtest/e2e/storage/testsuites/topology.go:194[0m

    [36mDriver didn't provide topology keys -- skipping[0m

    test/e2e/storage/testsuites/topology.go:126
[90m------------------------------[0m
... skipping 124 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":34,"completed":6,"skipped":1016,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow][0m 
  [1mshould concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS][0m
  [37mtest/e2e/storage/testsuites/multivolume.go:378[0m
... skipping 88 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:378[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":27,"completed":6,"skipped":472,"failed":0}

[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:34:48.729: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 122 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should store data
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data","total":28,"completed":8,"skipped":671,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m 
  [1mshould concurrently access the single read-only volume from pods on the same node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:423[0m
... skipping 88 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single read-only volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:423[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":44,"completed":8,"skipped":380,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould support existing directory[0m
  [37mtest/e2e/storage/testsuites/subpath.go:207[0m
... skipping 17 lines ...
Nov 30 02:34:49.717: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comp6g4g] to have phase Bound
Nov 30 02:34:49.826: INFO: PersistentVolumeClaim test.csi.azure.comp6g4g found but phase is Pending instead of Bound.
Nov 30 02:34:51.936: INFO: PersistentVolumeClaim test.csi.azure.comp6g4g found but phase is Pending instead of Bound.
Nov 30 02:34:54.046: INFO: PersistentVolumeClaim test.csi.azure.comp6g4g found and phase=Bound (4.328896002s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-k62q
[1mSTEP[0m: Creating a pod to test subpath
Nov 30 02:34:54.377: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-k62q" in namespace "provisioning-4655" to be "Succeeded or Failed"
Nov 30 02:34:54.486: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 108.917245ms
Nov 30 02:34:56.595: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218067986s
Nov 30 02:34:58.706: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328582749s
Nov 30 02:35:00.815: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437602581s
Nov 30 02:35:02.925: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547593393s
Nov 30 02:35:05.034: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 10.65687068s
... skipping 3 lines ...
Nov 30 02:35:13.489: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 19.11200462s
Nov 30 02:35:15.599: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 21.222057443s
Nov 30 02:35:17.711: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 23.333243709s
Nov 30 02:35:19.821: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Pending", Reason="", readiness=false. Elapsed: 25.444022952s
Nov 30 02:35:21.932: INFO: Pod "pod-subpath-test-dynamicpv-k62q": Phase="Succeeded", Reason="", readiness=false. Elapsed: 27.554180903s
[1mSTEP[0m: Saw pod success
Nov 30 02:35:21.932: INFO: Pod "pod-subpath-test-dynamicpv-k62q" satisfied condition "Succeeded or Failed"
Nov 30 02:35:22.040: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000001 pod pod-subpath-test-dynamicpv-k62q container test-container-volume-dynamicpv-k62q: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:35:22.308: INFO: Waiting for pod pod-subpath-test-dynamicpv-k62q to disappear
Nov 30 02:35:22.416: INFO: Pod pod-subpath-test-dynamicpv-k62q no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-k62q
Nov 30 02:35:22.416: INFO: Deleting pod "pod-subpath-test-dynamicpv-k62q" in namespace "provisioning-4655"
... skipping 23 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support existing directory
    [90mtest/e2e/storage/testsuites/subpath.go:207[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory","total":27,"completed":7,"skipped":476,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:36:04.333: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 55 lines ...

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:242[0m

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 30 02:32:51.143: INFO: >>> kubeConfig: /root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename provisioning
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  test/e2e/storage/testsuites/subpath.go:242
Nov 30 02:32:51.920: INFO: Creating resource for dynamic PV
Nov 30 02:32:51.920: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass provisioning-9366-e2e-sc5zs4m
[1mSTEP[0m: creating a claim
Nov 30 02:32:52.030: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Nov 30 02:32:52.142: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comqvqdt] to have phase Bound
Nov 30 02:32:52.251: INFO: PersistentVolumeClaim test.csi.azure.comqvqdt found but phase is Pending instead of Bound.
Nov 30 02:32:54.360: INFO: PersistentVolumeClaim test.csi.azure.comqvqdt found but phase is Pending instead of Bound.
Nov 30 02:32:56.470: INFO: PersistentVolumeClaim test.csi.azure.comqvqdt found and phase=Bound (4.328405641s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-dj5w
[1mSTEP[0m: Checking for subpath error in container status
Nov 30 02:33:57.040: INFO: Deleting pod "pod-subpath-test-dynamicpv-dj5w" in namespace "provisioning-9366"
Nov 30 02:33:57.151: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-dj5w" to be fully deleted
[1mSTEP[0m: Deleting pod
Nov 30 02:33:59.370: INFO: Deleting pod "pod-subpath-test-dynamicpv-dj5w" in namespace "provisioning-9366"
[1mSTEP[0m: Deleting pvc
Nov 30 02:33:59.480: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comqvqdt"
... skipping 34 lines ...

[32m• [SLOW TEST:202.012 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail if subpath directory is outside the volume [Slow][LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:242[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow][LinuxOnly]","total":30,"completed":5,"skipped":556,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m 
  [1mshould concurrently access the single read-only volume from pods on the same node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:423[0m
... skipping 88 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single read-only volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:423[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":44,"completed":9,"skipped":411,"failed":0}

[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:37:03.860: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 278 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:138[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":28,"completed":9,"skipped":680,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:37:14.759: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 278 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:168[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":36,"completed":6,"skipped":405,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould support existing directories when readOnly specified in the volumeSource[0m
  [37mtest/e2e/storage/testsuites/subpath.go:397[0m
... skipping 17 lines ...
Nov 30 02:33:28.331: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com852l4] to have phase Bound
Nov 30 02:33:28.442: INFO: PersistentVolumeClaim test.csi.azure.com852l4 found but phase is Pending instead of Bound.
Nov 30 02:33:30.552: INFO: PersistentVolumeClaim test.csi.azure.com852l4 found but phase is Pending instead of Bound.
Nov 30 02:33:32.660: INFO: PersistentVolumeClaim test.csi.azure.com852l4 found and phase=Bound (4.329563314s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-h9kj
[1mSTEP[0m: Creating a pod to test subpath
Nov 30 02:33:32.994: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-h9kj" in namespace "provisioning-2355" to be "Succeeded or Failed"
Nov 30 02:33:33.106: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 112.123643ms
Nov 30 02:33:35.220: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.22556396s
Nov 30 02:33:37.330: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.33586127s
Nov 30 02:33:39.439: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.445222526s
Nov 30 02:33:41.558: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.564138508s
Nov 30 02:33:43.669: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.674794711s
... skipping 29 lines ...
Nov 30 02:34:47.009: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.01538498s
Nov 30 02:34:49.119: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.125277142s
Nov 30 02:34:51.240: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.245518963s
Nov 30 02:34:53.350: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.355540398s
Nov 30 02:34:55.459: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m22.464408162s
[1mSTEP[0m: Saw pod success
Nov 30 02:34:55.459: INFO: Pod "pod-subpath-test-dynamicpv-h9kj" satisfied condition "Succeeded or Failed"
Nov 30 02:34:55.567: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000000 pod pod-subpath-test-dynamicpv-h9kj container test-container-subpath-dynamicpv-h9kj: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:34:55.857: INFO: Waiting for pod pod-subpath-test-dynamicpv-h9kj to disappear
Nov 30 02:34:55.971: INFO: Pod pod-subpath-test-dynamicpv-h9kj no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-h9kj
Nov 30 02:34:55.971: INFO: Deleting pod "pod-subpath-test-dynamicpv-h9kj" in namespace "provisioning-2355"
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-h9kj
[1mSTEP[0m: Creating a pod to test subpath
Nov 30 02:34:56.200: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-h9kj" in namespace "provisioning-2355" to be "Succeeded or Failed"
Nov 30 02:34:56.316: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 115.829706ms
Nov 30 02:34:58.425: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.225325925s
Nov 30 02:35:00.535: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.334637018s
Nov 30 02:35:02.644: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.444214751s
Nov 30 02:35:04.754: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.554437495s
Nov 30 02:35:06.865: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.665239927s
... skipping 37 lines ...
Nov 30 02:36:27.050: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m30.850259726s
Nov 30 02:36:29.160: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m32.960426324s
Nov 30 02:36:31.272: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m35.071541826s
Nov 30 02:36:33.381: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Pending", Reason="", readiness=false. Elapsed: 1m37.180918481s
Nov 30 02:36:35.492: INFO: Pod "pod-subpath-test-dynamicpv-h9kj": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m39.292046642s
[1mSTEP[0m: Saw pod success
Nov 30 02:36:35.492: INFO: Pod "pod-subpath-test-dynamicpv-h9kj" satisfied condition "Succeeded or Failed"
Nov 30 02:36:35.601: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000000 pod pod-subpath-test-dynamicpv-h9kj container test-container-subpath-dynamicpv-h9kj: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:36:35.856: INFO: Waiting for pod pod-subpath-test-dynamicpv-h9kj to disappear
Nov 30 02:36:35.965: INFO: Pod pod-subpath-test-dynamicpv-h9kj no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-h9kj
Nov 30 02:36:35.965: INFO: Deleting pod "pod-subpath-test-dynamicpv-h9kj" in namespace "provisioning-2355"
... skipping 29 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support existing directories when readOnly specified in the volumeSource
    [90mtest/e2e/storage/testsuites/subpath.go:397[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource","total":31,"completed":6,"skipped":403,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:37:48.535: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 172 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:298[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":27,"completed":8,"skipped":630,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] volumeMode[0m 
  [1mshould not mount / map unused volumes in a pod [LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/volumemode.go:354[0m
... skipping 69 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should not mount / map unused volumes in a pod [LinuxOnly]
    [90mtest/e2e/storage/testsuites/volumemode.go:354[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":44,"completed":10,"skipped":584,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:38:31.746: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/volumemode.go:299[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 191 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:298[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":28,"completed":10,"skipped":901,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow][0m 
  [1mshould access to two volumes with different volume mode and retain data across pod recreation on different node[0m
  [37mtest/e2e/storage/testsuites/multivolume.go:248[0m
... skipping 195 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:248[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":30,"completed":6,"skipped":658,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 55 lines ...
Nov 30 02:38:15.239: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comw22jm] to have phase Bound
Nov 30 02:38:15.348: INFO: PersistentVolumeClaim test.csi.azure.comw22jm found but phase is Pending instead of Bound.
Nov 30 02:38:17.457: INFO: PersistentVolumeClaim test.csi.azure.comw22jm found but phase is Pending instead of Bound.
Nov 30 02:38:19.570: INFO: PersistentVolumeClaim test.csi.azure.comw22jm found and phase=Bound (4.331249252s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-2p42
[1mSTEP[0m: Creating a pod to test multi_subpath
Nov 30 02:38:19.897: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-2p42" in namespace "provisioning-2464" to be "Succeeded or Failed"
Nov 30 02:38:20.005: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 108.117303ms
Nov 30 02:38:22.114: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21721424s
Nov 30 02:38:24.224: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327527847s
Nov 30 02:38:26.334: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436964992s
Nov 30 02:38:28.444: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 8.546804995s
Nov 30 02:38:30.552: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655243424s
... skipping 25 lines ...
Nov 30 02:39:25.405: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.508620713s
Nov 30 02:39:27.515: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.61828019s
Nov 30 02:39:29.624: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.727490304s
Nov 30 02:39:31.734: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.837596112s
Nov 30 02:39:33.849: INFO: Pod "pod-subpath-test-dynamicpv-2p42": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m13.951835453s
[1mSTEP[0m: Saw pod success
Nov 30 02:39:33.849: INFO: Pod "pod-subpath-test-dynamicpv-2p42" satisfied condition "Succeeded or Failed"
Nov 30 02:39:33.958: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000001 pod pod-subpath-test-dynamicpv-2p42 container test-container-subpath-dynamicpv-2p42: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:39:34.186: INFO: Waiting for pod pod-subpath-test-dynamicpv-2p42 to disappear
Nov 30 02:39:34.295: INFO: Pod pod-subpath-test-dynamicpv-2p42 no longer exists
[1mSTEP[0m: Deleting pod
Nov 30 02:39:34.295: INFO: Deleting pod "pod-subpath-test-dynamicpv-2p42" in namespace "provisioning-2464"
... skipping 27 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support creating multiple subpath from same volumes [Slow]
    [90mtest/e2e/storage/testsuites/subpath.go:296[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support creating multiple subpath from same volumes [Slow]","total":27,"completed":9,"skipped":659,"failed":0}

[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource]
... skipping 111 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read-only inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:175[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume","total":44,"completed":11,"skipped":715,"failed":0}

[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] volumeIO[0m 
  [1mshould write files of various sizes, verify size, validate content [Slow][0m
  [37mtest/e2e/storage/testsuites/volume_io.go:149[0m
... skipping 58 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] volumeIO
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should write files of various sizes, verify size, validate content [Slow]
    [90mtest/e2e/storage/testsuites/volume_io.go:149[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow]","total":28,"completed":11,"skipped":930,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:41:49.210: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:269[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 32 lines ...
Nov 30 02:37:49.550: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comr2vbg] to have phase Bound
Nov 30 02:37:49.658: INFO: PersistentVolumeClaim test.csi.azure.comr2vbg found but phase is Pending instead of Bound.
Nov 30 02:37:51.768: INFO: PersistentVolumeClaim test.csi.azure.comr2vbg found but phase is Pending instead of Bound.
Nov 30 02:37:53.904: INFO: PersistentVolumeClaim test.csi.azure.comr2vbg found and phase=Bound (4.353875551s)
[1mSTEP[0m: [init] starting a pod to use the claim
[1mSTEP[0m: [init] check pod success
Nov 30 02:37:54.341: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-5ccwl" in namespace "snapshotting-1502" to be "Succeeded or Failed"
Nov 30 02:37:54.450: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 108.787243ms
Nov 30 02:37:56.561: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219438588s
Nov 30 02:37:58.670: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329213741s
Nov 30 02:38:00.779: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438157805s
Nov 30 02:38:02.889: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 8.548185116s
Nov 30 02:38:04.999: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 10.658015703s
Nov 30 02:38:07.109: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 12.768231424s
Nov 30 02:38:09.219: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 14.877992457s
Nov 30 02:38:11.328: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 16.986512367s
Nov 30 02:38:13.438: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Pending", Reason="", readiness=false. Elapsed: 19.096659685s
Nov 30 02:38:15.547: INFO: Pod "pvc-snapshottable-tester-5ccwl": Phase="Succeeded", Reason="", readiness=false. Elapsed: 21.206357523s
[1mSTEP[0m: Saw pod success
Nov 30 02:38:15.548: INFO: Pod "pvc-snapshottable-tester-5ccwl" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
Nov 30 02:38:15.656: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comr2vbg] to have phase Bound
Nov 30 02:38:15.765: INFO: PersistentVolumeClaim test.csi.azure.comr2vbg found and phase=Bound (108.553265ms)
[1mSTEP[0m: [init] checking the PV
[1mSTEP[0m: [init] deleting the pod
Nov 30 02:38:16.120: INFO: Pod pvc-snapshottable-tester-5ccwl has the following logs: 
... skipping 33 lines ...
Nov 30 02:38:24.620: INFO: WaitUntil finished successfully after 152.75757ms
[1mSTEP[0m: getting the snapshot and snapshot content
[1mSTEP[0m: checking the snapshot
[1mSTEP[0m: checking the SnapshotContent
[1mSTEP[0m: Modifying source data test
[1mSTEP[0m: modifying the data in the source PVC
Nov 30 02:38:25.169: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-6jng4" in namespace "snapshotting-1502" to be "Succeeded or Failed"
Nov 30 02:38:25.284: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 115.326253ms
Nov 30 02:38:27.395: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.225877687s
Nov 30 02:38:29.506: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.337089025s
Nov 30 02:38:31.618: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.449342504s
Nov 30 02:38:33.729: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.559872482s
Nov 30 02:38:35.838: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.669699889s
... skipping 29 lines ...
Nov 30 02:39:39.146: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.977034916s
Nov 30 02:39:41.259: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.090733638s
Nov 30 02:39:43.370: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.201710432s
Nov 30 02:39:45.482: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.313183942s
Nov 30 02:39:47.593: INFO: Pod "pvc-snapshottable-data-tester-6jng4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m22.42412807s
[1mSTEP[0m: Saw pod success
Nov 30 02:39:47.593: INFO: Pod "pvc-snapshottable-data-tester-6jng4" satisfied condition "Succeeded or Failed"
Nov 30 02:39:47.818: INFO: Pod pvc-snapshottable-data-tester-6jng4 has the following logs: 
Nov 30 02:39:47.818: INFO: Deleting pod "pvc-snapshottable-data-tester-6jng4" in namespace "snapshotting-1502"
Nov 30 02:39:47.933: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-6jng4" to be fully deleted
[1mSTEP[0m: creating a pvc from the snapshot
[1mSTEP[0m: starting a pod to use the snapshot
Nov 30 02:40:14.491: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-5bbt3k3k.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-1502 exec restored-pvc-tester-9ghd4 --namespace=snapshotting-1502 -- cat /mnt/test/data'
... skipping 32 lines ...
Nov 30 02:40:39.709: INFO: volumesnapshotcontents pre-provisioned-snapcontent-d59d1f22-e995-4891-88be-398e556ff789 has been found and is not deleted
Nov 30 02:40:40.821: INFO: volumesnapshotcontents pre-provisioned-snapcontent-d59d1f22-e995-4891-88be-398e556ff789 has been found and is not deleted
Nov 30 02:40:41.931: INFO: volumesnapshotcontents pre-provisioned-snapcontent-d59d1f22-e995-4891-88be-398e556ff789 has been found and is not deleted
Nov 30 02:40:43.042: INFO: volumesnapshotcontents pre-provisioned-snapcontent-d59d1f22-e995-4891-88be-398e556ff789 has been found and is not deleted
Nov 30 02:40:44.157: INFO: volumesnapshotcontents pre-provisioned-snapcontent-d59d1f22-e995-4891-88be-398e556ff789 has been found and is not deleted
Nov 30 02:40:45.268: INFO: volumesnapshotcontents pre-provisioned-snapcontent-d59d1f22-e995-4891-88be-398e556ff789 has been found and is not deleted
Nov 30 02:40:46.269: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 30 02:40:46.425: INFO: Pod restored-pvc-tester-9ghd4 has the following logs: 
Nov 30 02:40:46.425: INFO: Deleting pod "restored-pvc-tester-9ghd4" in namespace "snapshotting-1502"
Nov 30 02:40:46.534: INFO: Wait up to 5m0s for pod "restored-pvc-tester-9ghd4" to be fully deleted
Nov 30 02:40:48.756: INFO: deleting claim "snapshotting-1502"/"pvc-jnrpl"
... skipping 43 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":31,"completed":7,"skipped":444,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
... skipping 152 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:323[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":36,"completed":7,"skipped":446,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:43:40.886: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 24 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:269[0m

    [36mDistro debian doesn't support ntfs -- skipping[0m

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
... skipping 77 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support multiple inline ephemeral volumes
    [90mtest/e2e/storage/testsuites/ephemeral.go:254[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":30,"completed":7,"skipped":701,"failed":0}

[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould support readOnly file specified in the volumeMount [LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:382[0m
... skipping 17 lines ...
Nov 30 02:41:35.639: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comnd26h] to have phase Bound
Nov 30 02:41:35.747: INFO: PersistentVolumeClaim test.csi.azure.comnd26h found but phase is Pending instead of Bound.
Nov 30 02:41:37.857: INFO: PersistentVolumeClaim test.csi.azure.comnd26h found but phase is Pending instead of Bound.
Nov 30 02:41:39.966: INFO: PersistentVolumeClaim test.csi.azure.comnd26h found and phase=Bound (4.326850814s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-ldrx
[1mSTEP[0m: Creating a pod to test subpath
Nov 30 02:41:40.302: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-ldrx" in namespace "provisioning-6249" to be "Succeeded or Failed"
Nov 30 02:41:40.410: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 108.542302ms
Nov 30 02:41:42.521: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218736271s
Nov 30 02:41:44.631: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 4.32882283s
Nov 30 02:41:46.741: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438669468s
Nov 30 02:41:48.850: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547862975s
Nov 30 02:41:50.960: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657713307s
... skipping 11 lines ...
Nov 30 02:42:16.281: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 35.979322828s
Nov 30 02:42:18.392: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 38.08976349s
Nov 30 02:42:20.502: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 40.200380549s
Nov 30 02:42:22.613: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Pending", Reason="", readiness=false. Elapsed: 42.310643594s
Nov 30 02:42:24.722: INFO: Pod "pod-subpath-test-dynamicpv-ldrx": Phase="Succeeded", Reason="", readiness=false. Elapsed: 44.419624756s
[1mSTEP[0m: Saw pod success
Nov 30 02:42:24.722: INFO: Pod "pod-subpath-test-dynamicpv-ldrx" satisfied condition "Succeeded or Failed"
Nov 30 02:42:24.830: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000000 pod pod-subpath-test-dynamicpv-ldrx container test-container-subpath-dynamicpv-ldrx: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:42:25.057: INFO: Waiting for pod pod-subpath-test-dynamicpv-ldrx to disappear
Nov 30 02:42:25.166: INFO: Pod pod-subpath-test-dynamicpv-ldrx no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-ldrx
Nov 30 02:42:25.166: INFO: Deleting pod "pod-subpath-test-dynamicpv-ldrx" in namespace "provisioning-6249"
... skipping 40 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support readOnly file specified in the volumeMount [LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:382[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]","total":44,"completed":12,"skipped":718,"failed":0}

[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] volumeMode[0m 
  [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m
  [37mtest/e2e/storage/testsuites/volumemode.go:299[0m

[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 30 02:44:07.528: INFO: >>> kubeConfig: /root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename volumemode
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 30 02:44:08.291: INFO: Creating resource for dynamic PV
Nov 30 02:44:08.291: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass volumemode-588-e2e-scghhj6
[1mSTEP[0m: creating a claim
Nov 30 02:44:08.511: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com8bkw8] to have phase Bound
Nov 30 02:44:08.619: INFO: PersistentVolumeClaim test.csi.azure.com8bkw8 found but phase is Pending instead of Bound.
Nov 30 02:44:10.728: INFO: PersistentVolumeClaim test.csi.azure.com8bkw8 found but phase is Pending instead of Bound.
Nov 30 02:44:12.838: INFO: PersistentVolumeClaim test.csi.azure.com8bkw8 found and phase=Bound (4.326582426s)
[1mSTEP[0m: Creating pod
[1mSTEP[0m: Waiting for the pod to fail
Nov 30 02:44:15.493: INFO: Deleting pod "pod-4959f7c6-2f2e-4109-bdbb-b17b24572166" in namespace "volumemode-588"
Nov 30 02:44:15.604: INFO: Wait up to 5m0s for pod "pod-4959f7c6-2f2e-4109-bdbb-b17b24572166" to be fully deleted
[1mSTEP[0m: Deleting pvc
Nov 30 02:44:17.824: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com8bkw8"
Nov 30 02:44:17.936: INFO: Waiting up to 5m0s for PersistentVolume pvc-2a648284-4cad-43d2-a719-8a9b0acbf7ad to get deleted
Nov 30 02:44:18.044: INFO: PersistentVolume pvc-2a648284-4cad-43d2-a719-8a9b0acbf7ad found and phase=Released (107.926615ms)
... skipping 20 lines ...

[32m• [SLOW TEST:82.602 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail to use a volume in a pod with mismatched mode [Slow]
    [90mtest/e2e/storage/testsuites/volumemode.go:299[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":30,"completed":8,"skipped":702,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:45:30.187: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:258[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 22 lines ...
Nov 30 02:40:48.896: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comxqngk] to have phase Bound
Nov 30 02:40:49.004: INFO: PersistentVolumeClaim test.csi.azure.comxqngk found but phase is Pending instead of Bound.
Nov 30 02:40:51.112: INFO: PersistentVolumeClaim test.csi.azure.comxqngk found but phase is Pending instead of Bound.
Nov 30 02:40:53.221: INFO: PersistentVolumeClaim test.csi.azure.comxqngk found and phase=Bound (4.325501591s)
[1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-jvsh
[1mSTEP[0m: Creating a pod to test exec-volume-test
Nov 30 02:40:53.548: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-jvsh" in namespace "volume-5636" to be "Succeeded or Failed"
Nov 30 02:40:53.656: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 107.71741ms
Nov 30 02:40:55.766: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217397086s
Nov 30 02:40:57.876: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327505222s
Nov 30 02:40:59.986: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437358383s
Nov 30 02:41:02.096: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547798621s
Nov 30 02:41:04.206: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657604569s
... skipping 19 lines ...
Nov 30 02:41:46.406: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 52.857994583s
Nov 30 02:41:48.517: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 54.968397286s
Nov 30 02:41:50.626: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 57.077784452s
Nov 30 02:41:52.736: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Pending", Reason="", readiness=false. Elapsed: 59.187576398s
Nov 30 02:41:54.855: INFO: Pod "exec-volume-test-dynamicpv-jvsh": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m1.30646112s
[1mSTEP[0m: Saw pod success
Nov 30 02:41:54.855: INFO: Pod "exec-volume-test-dynamicpv-jvsh" satisfied condition "Succeeded or Failed"
Nov 30 02:41:54.970: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000000 pod exec-volume-test-dynamicpv-jvsh container exec-container-dynamicpv-jvsh: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:41:55.227: INFO: Waiting for pod exec-volume-test-dynamicpv-jvsh to disappear
Nov 30 02:41:55.335: INFO: Pod exec-volume-test-dynamicpv-jvsh no longer exists
[1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-jvsh
Nov 30 02:41:55.335: INFO: Deleting pod "exec-volume-test-dynamicpv-jvsh" in namespace "volume-5636"
... skipping 57 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext3)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should allow exec of files on the volume
    [90mtest/e2e/storage/testsuites/volumes.go:198[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume","total":27,"completed":10,"skipped":783,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource][0m [0mvolume snapshot controller[0m [90m[0m 
  [1mshould check snapshot fields, check restore correctly works, check deletion (ephemeral)[0m
  [37mtest/e2e/storage/testsuites/snapshottable.go:177[0m
... skipping 10 lines ...
[It] should check snapshot fields, check restore correctly works, check deletion (ephemeral)
  test/e2e/storage/testsuites/snapshottable.go:177
Nov 30 02:43:41.723: INFO: Creating resource for dynamic PV
Nov 30 02:43:41.723: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass snapshotting-4875-e2e-sc2vff7
[1mSTEP[0m: [init] starting a pod to use the claim
Nov 30 02:43:41.949: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-hq88b" in namespace "snapshotting-4875" to be "Succeeded or Failed"
Nov 30 02:43:42.065: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 115.571201ms
Nov 30 02:43:44.175: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 2.225535177s
Nov 30 02:43:46.288: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 4.339112602s
Nov 30 02:43:48.399: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 6.449716367s
Nov 30 02:43:50.509: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 8.559517497s
Nov 30 02:43:52.619: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 10.669404944s
Nov 30 02:43:54.728: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 12.778888567s
Nov 30 02:43:56.840: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 14.890562575s
Nov 30 02:43:58.951: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 17.001369299s
Nov 30 02:44:01.060: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Pending", Reason="", readiness=false. Elapsed: 19.111060167s
Nov 30 02:44:03.170: INFO: Pod "pvc-snapshottable-tester-hq88b": Phase="Succeeded", Reason="", readiness=false. Elapsed: 21.220646904s
[1mSTEP[0m: Saw pod success
Nov 30 02:44:03.170: INFO: Pod "pvc-snapshottable-tester-hq88b" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
[1mSTEP[0m: creating a SnapshotClass
[1mSTEP[0m: creating a dynamic VolumeSnapshot
Nov 30 02:44:03.611: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-5x6vn to become ready
Nov 30 02:44:03.720: INFO: VolumeSnapshot snapshot-5x6vn found but is not ready.
Nov 30 02:44:05.830: INFO: VolumeSnapshot snapshot-5x6vn found but is not ready.
... skipping 49 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works, check deletion (ephemeral)
        [90mtest/e2e/storage/testsuites/snapshottable.go:177[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":36,"completed":8,"skipped":506,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral[0m 
  [1mshould support two pods which have the same volume definition[0m
  [37mtest/e2e/storage/testsuites/ephemeral.go:216[0m
... skipping 70 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support two pods which have the same volume definition
    [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition","total":31,"completed":8,"skipped":591,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy[0m 
  [1m(OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents[0m
  [37mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
... skipping 97 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":44,"completed":13,"skipped":720,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould be able to unmount after the subpath directory is deleted [LinuxOnly][0m
  [37mtest/e2e/storage/testsuites/subpath.go:447[0m
... skipping 74 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should be able to unmount after the subpath directory is deleted [LinuxOnly]
    [90mtest/e2e/storage/testsuites/subpath.go:447[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]","total":36,"completed":9,"skipped":557,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (ext4)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:49:08.192: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 59 lines ...
Nov 30 02:45:42.162: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comqpvp7] to have phase Bound
Nov 30 02:45:42.271: INFO: PersistentVolumeClaim test.csi.azure.comqpvp7 found but phase is Pending instead of Bound.
Nov 30 02:45:44.387: INFO: PersistentVolumeClaim test.csi.azure.comqpvp7 found but phase is Pending instead of Bound.
Nov 30 02:45:46.495: INFO: PersistentVolumeClaim test.csi.azure.comqpvp7 found and phase=Bound (4.333133906s)
[1mSTEP[0m: [init] starting a pod to use the claim
[1mSTEP[0m: [init] check pod success
Nov 30 02:45:46.939: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-b6fch" in namespace "snapshotting-5488" to be "Succeeded or Failed"
Nov 30 02:45:47.047: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 108.583841ms
Nov 30 02:45:49.157: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218200716s
Nov 30 02:45:51.268: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329074938s
Nov 30 02:45:53.377: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438683178s
Nov 30 02:45:55.486: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547395126s
Nov 30 02:45:57.595: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 10.656451981s
Nov 30 02:45:59.704: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 12.765685357s
Nov 30 02:46:01.814: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 14.875072303s
Nov 30 02:46:03.923: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 16.984471672s
Nov 30 02:46:06.032: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Pending", Reason="", readiness=false. Elapsed: 19.093491792s
Nov 30 02:46:08.143: INFO: Pod "pvc-snapshottable-tester-b6fch": Phase="Succeeded", Reason="", readiness=false. Elapsed: 21.204429307s
[1mSTEP[0m: Saw pod success
Nov 30 02:46:08.143: INFO: Pod "pvc-snapshottable-tester-b6fch" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
Nov 30 02:46:08.252: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comqpvp7] to have phase Bound
Nov 30 02:46:08.361: INFO: PersistentVolumeClaim test.csi.azure.comqpvp7 found and phase=Bound (108.935308ms)
[1mSTEP[0m: [init] checking the PV
[1mSTEP[0m: [init] deleting the pod
Nov 30 02:46:08.690: INFO: Pod pvc-snapshottable-tester-b6fch has the following logs: 
... skipping 33 lines ...
Nov 30 02:46:17.122: INFO: WaitUntil finished successfully after 111.780452ms
[1mSTEP[0m: getting the snapshot and snapshot content
[1mSTEP[0m: checking the snapshot
[1mSTEP[0m: checking the SnapshotContent
[1mSTEP[0m: Modifying source data test
[1mSTEP[0m: modifying the data in the source PVC
Nov 30 02:46:17.671: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-dw4nd" in namespace "snapshotting-5488" to be "Succeeded or Failed"
Nov 30 02:46:17.779: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 108.379511ms
Nov 30 02:46:19.889: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217694591s
Nov 30 02:46:21.998: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327138958s
Nov 30 02:46:24.108: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437295992s
Nov 30 02:46:26.218: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547200139s
Nov 30 02:46:28.328: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657295485s
... skipping 16 lines ...
Nov 30 02:47:04.211: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 46.540341168s
Nov 30 02:47:06.321: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 48.650006061s
Nov 30 02:47:08.431: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 50.760353432s
Nov 30 02:47:10.541: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Pending", Reason="", readiness=false. Elapsed: 52.870022356s
Nov 30 02:47:12.652: INFO: Pod "pvc-snapshottable-data-tester-dw4nd": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.980865196s
[1mSTEP[0m: Saw pod success
Nov 30 02:47:12.652: INFO: Pod "pvc-snapshottable-data-tester-dw4nd" satisfied condition "Succeeded or Failed"
Nov 30 02:47:12.879: INFO: Pod pvc-snapshottable-data-tester-dw4nd has the following logs: 
Nov 30 02:47:12.879: INFO: Deleting pod "pvc-snapshottable-data-tester-dw4nd" in namespace "snapshotting-5488"
Nov 30 02:47:12.998: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-dw4nd" to be fully deleted
[1mSTEP[0m: creating a pvc from the snapshot
[1mSTEP[0m: starting a pod to use the snapshot
Nov 30 02:49:49.552: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-5bbt3k3k.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-5488 exec restored-pvc-tester-ls2dv --namespace=snapshotting-5488 -- cat /mnt/test/data'
... skipping 53 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":27,"completed":11,"skipped":852,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy[0m 
  [1m(Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents[0m
  [37mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
... skipping 97 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents
    [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":36,"completed":10,"skipped":579,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] provisioning[0m 
  [1mshould provision storage with pvc data source in parallel [Slow][0m
  [37mtest/e2e/storage/testsuites/provisioning.go:459[0m
... skipping 388 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with pvc data source in parallel [Slow]
    [90mtest/e2e/storage/testsuites/provisioning.go:459[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source in parallel [Slow]","total":30,"completed":9,"skipped":767,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral
... skipping 229 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:298[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":31,"completed":9,"skipped":597,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology
... skipping 200 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support restarting containers using directory as subpath [Slow]
    [90mtest/e2e/storage/testsuites/subpath.go:322[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]","total":44,"completed":14,"skipped":727,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (ntfs)][Feature:Windows] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:53:30.269: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 73 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read-only inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:175[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume","total":36,"completed":11,"skipped":638,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (default fs)] volume-expand
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:54:39.412: INFO: Driver "test.csi.azure.com" does not support volume expansion - skipping
... skipping 215 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:209[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":27,"completed":12,"skipped":991,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:54:45.950: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 38 lines ...
Nov 30 02:53:31.269: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comf248g] to have phase Bound
Nov 30 02:53:31.377: INFO: PersistentVolumeClaim test.csi.azure.comf248g found but phase is Pending instead of Bound.
Nov 30 02:53:33.486: INFO: PersistentVolumeClaim test.csi.azure.comf248g found but phase is Pending instead of Bound.
Nov 30 02:53:35.599: INFO: PersistentVolumeClaim test.csi.azure.comf248g found and phase=Bound (4.329944061s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-nvlc
[1mSTEP[0m: Creating a pod to test subpath
Nov 30 02:53:35.927: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-nvlc" in namespace "provisioning-4200" to be "Succeeded or Failed"
Nov 30 02:53:36.036: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 108.556558ms
Nov 30 02:53:38.146: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219082008s
Nov 30 02:53:40.260: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 4.332828516s
Nov 30 02:53:42.370: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 6.44295349s
Nov 30 02:53:44.482: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 8.554208242s
Nov 30 02:53:46.592: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 10.664347141s
... skipping 14 lines ...
Nov 30 02:54:18.243: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 42.315852864s
Nov 30 02:54:20.353: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 44.425115171s
Nov 30 02:54:22.462: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 46.534895061s
Nov 30 02:54:24.572: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Pending", Reason="", readiness=false. Elapsed: 48.644716689s
Nov 30 02:54:26.683: INFO: Pod "pod-subpath-test-dynamicpv-nvlc": Phase="Succeeded", Reason="", readiness=false. Elapsed: 50.755271836s
[1mSTEP[0m: Saw pod success
Nov 30 02:54:26.683: INFO: Pod "pod-subpath-test-dynamicpv-nvlc" satisfied condition "Succeeded or Failed"
Nov 30 02:54:26.792: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000000 pod pod-subpath-test-dynamicpv-nvlc container test-container-subpath-dynamicpv-nvlc: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:54:27.049: INFO: Waiting for pod pod-subpath-test-dynamicpv-nvlc to disappear
Nov 30 02:54:27.158: INFO: Pod pod-subpath-test-dynamicpv-nvlc no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-nvlc
Nov 30 02:54:27.158: INFO: Deleting pod "pod-subpath-test-dynamicpv-nvlc" in namespace "provisioning-4200"
... skipping 23 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should support readOnly directory specified in the volumeMount
    [90mtest/e2e/storage/testsuites/subpath.go:367[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount","total":44,"completed":15,"skipped":755,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:55:09.032: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 66 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:258[0m

    [36mDistro debian doesn't support ntfs -- skipping[0m

    test/e2e/storage/framework/testsuite.go:127
[90m------------------------------[0m
... skipping 64 lines ...
Nov 30 02:54:40.414: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com2stgq] to have phase Bound
Nov 30 02:54:40.525: INFO: PersistentVolumeClaim test.csi.azure.com2stgq found but phase is Pending instead of Bound.
Nov 30 02:54:42.634: INFO: PersistentVolumeClaim test.csi.azure.com2stgq found but phase is Pending instead of Bound.
Nov 30 02:54:44.745: INFO: PersistentVolumeClaim test.csi.azure.com2stgq found and phase=Bound (4.330680531s)
[1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-6zfb
[1mSTEP[0m: Creating a pod to test exec-volume-test
Nov 30 02:54:45.076: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-6zfb" in namespace "volume-5654" to be "Succeeded or Failed"
Nov 30 02:54:45.194: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 118.00181ms
Nov 30 02:54:47.306: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 2.230083395s
Nov 30 02:54:49.421: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 4.344631855s
Nov 30 02:54:51.531: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 6.454678114s
Nov 30 02:54:53.642: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 8.565632374s
Nov 30 02:54:55.753: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 10.676773505s
... skipping 13 lines ...
Nov 30 02:55:25.307: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 40.230957335s
Nov 30 02:55:27.418: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 42.341552731s
Nov 30 02:55:29.528: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 44.451720324s
Nov 30 02:55:31.639: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Pending", Reason="", readiness=false. Elapsed: 46.56296567s
Nov 30 02:55:33.750: INFO: Pod "exec-volume-test-dynamicpv-6zfb": Phase="Succeeded", Reason="", readiness=false. Elapsed: 48.67425075s
[1mSTEP[0m: Saw pod success
Nov 30 02:55:33.750: INFO: Pod "exec-volume-test-dynamicpv-6zfb" satisfied condition "Succeeded or Failed"
Nov 30 02:55:33.870: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000000 pod exec-volume-test-dynamicpv-6zfb container exec-container-dynamicpv-6zfb: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:55:34.104: INFO: Waiting for pod exec-volume-test-dynamicpv-6zfb to disappear
Nov 30 02:55:34.215: INFO: Pod exec-volume-test-dynamicpv-6zfb no longer exists
[1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-6zfb
Nov 30 02:55:34.215: INFO: Deleting pod "exec-volume-test-dynamicpv-6zfb" in namespace "volume-5654"
... skipping 36 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should allow exec of files on the volume
    [90mtest/e2e/storage/testsuites/volumes.go:198[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should allow exec of files on the volume","total":36,"completed":12,"skipped":802,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:57:32.699: INFO: Distro debian doesn't support ntfs -- skipping
... skipping 215 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with different volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:248[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":31,"completed":10,"skipped":707,"failed":0}

[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:57:43.037: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 120 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single read-only volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:423[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":44,"completed":16,"skipped":867,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 02:59:47.980: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 3 lines ...

[36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m
    [90mtest/e2e/storage/testsuites/subpath.go:242[0m

    [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m

    test/e2e/storage/external/external.go:262
[90m------------------------------[0m
... skipping 149 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]
    [90mtest/e2e/storage/testsuites/multivolume.go:378[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":30,"completed":10,"skipped":951,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] provisioning[0m 
  [1mshould provision storage with pvc data source[0m
  [37mtest/e2e/storage/testsuites/provisioning.go:421[0m
... skipping 107 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with pvc data source
    [90mtest/e2e/storage/testsuites/provisioning.go:421[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source","total":36,"completed":13,"skipped":845,"failed":0}

[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m 
  [1mshould support non-existent path[0m
  [37mtest/e2e/storage/testsuites/subpath.go:196[0m
... skipping 17 lines ...
Nov 30 02:54:46.945: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com9frnp] to have phase Bound
Nov 30 02:54:47.052: INFO: PersistentVolumeClaim test.csi.azure.com9frnp found but phase is Pending instead of Bound.
Nov 30 02:54:49.161: INFO: PersistentVolumeClaim test.csi.azure.com9frnp found but phase is Pending instead of Bound.
Nov 30 02:54:51.270: INFO: PersistentVolumeClaim test.csi.azure.com9frnp found and phase=Bound (4.325540659s)
[1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-wwt6
[1mSTEP[0m: Creating a pod to test subpath
Nov 30 02:54:51.597: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-wwt6" in namespace "provisioning-5526" to be "Succeeded or Failed"
Nov 30 02:54:51.713: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 115.85174ms
Nov 30 02:54:53.823: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.225254436s
Nov 30 02:54:55.931: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.334043883s
Nov 30 02:54:58.040: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.442881839s
Nov 30 02:55:00.149: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551937709s
Nov 30 02:55:02.260: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.662534326s
... skipping 26 lines ...
Nov 30 02:55:59.240: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.642232444s
Nov 30 02:56:01.349: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.751628867s
Nov 30 02:56:03.459: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.861642823s
Nov 30 02:56:05.569: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.972023878s
Nov 30 02:56:07.680: INFO: Pod "pod-subpath-test-dynamicpv-wwt6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m16.08259332s
[1mSTEP[0m: Saw pod success
Nov 30 02:56:07.680: INFO: Pod "pod-subpath-test-dynamicpv-wwt6" satisfied condition "Succeeded or Failed"
Nov 30 02:56:07.788: INFO: Trying to get logs from node k8s-agentpool1-10232909-vmss000001 pod pod-subpath-test-dynamicpv-wwt6 container test-container-volume-dynamicpv-wwt6: <nil>
[1mSTEP[0m: delete the pod
Nov 30 02:56:08.043: INFO: Waiting for pod pod-subpath-test-dynamicpv-wwt6 to disappear
Nov 30 02:56:08.151: INFO: Pod pod-subpath-test-dynamicpv-wwt6 no longer exists
[1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-wwt6
Nov 30 02:56:08.151: INFO: Deleting pod "pod-subpath-test-dynamicpv-wwt6" in namespace "provisioning-5526"
... skipping 59 lines ...
Nov 30 03:00:44.574: INFO: PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d found and phase=Released (4m36.087979053s)
Nov 30 03:00:49.684: INFO: PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d found and phase=Released (4m41.197675034s)
Nov 30 03:00:54.792: INFO: PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d found and phase=Released (4m46.306185547s)
Nov 30 03:00:59.901: INFO: PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d found and phase=Released (4m51.415041517s)
Nov 30 03:01:05.022: INFO: PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d found and phase=Released (4m56.536465205s)
[1mSTEP[0m: Deleting sc
Nov 30 03:01:10.134: FAIL: while cleaning up resource
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        [
            {
                msg: "persistent Volume pvc-42b83320-2e34-4e17-a558-43ec0e08855d not deleted by dynamic provisioner: PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d still exists within 5m0s",
                err: {
                    s: "PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d still exists within 5m0s",
... skipping 160 lines ...
  [Testpattern: Dynamic PV (default fs)] subPath
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [91m[1mshould support non-existent path [Measurement][0m
    [90mtest/e2e/storage/testsuites/subpath.go:196[0m

    [91mNov 30 03:01:10.134: while cleaning up resource
    Unexpected error:
        <errors.aggregate | len:1, cap:1>: [
            [
                {
                    msg: "persistent Volume pvc-42b83320-2e34-4e17-a558-43ec0e08855d not deleted by dynamic provisioner: PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d still exists within 5m0s",
                    err: {
                        s: "PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d still exists within 5m0s",
... skipping 3 lines ...
        ]
        persistent Volume pvc-42b83320-2e34-4e17-a558-43ec0e08855d not deleted by dynamic provisioner: PersistentVolume pvc-42b83320-2e34-4e17-a558-43ec0e08855d still exists within 5m0s
    occurred[0m

    test/e2e/storage/testsuites/subpath.go:185
[90m------------------------------[0m
{"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path","total":27,"completed":12,"skipped":1007,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path"]}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
Nov 30 03:01:13.905: INFO: Running AfterSuite actions on all nodes
Nov 30 03:01:13.905: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 30 03:01:13.905: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 92 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should concurrently access the single read-only volume from pods on the same node
    [90mtest/e2e/storage/testsuites/multivolume.go:423[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":30,"completed":11,"skipped":956,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  test/e2e/storage/framework/testsuite.go:51
Nov 30 03:01:27.056: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 232 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:168[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":31,"completed":11,"skipped":788,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  test/e2e/storage/framework/testsuite.go:51
Nov 30 03:01:40.460: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 150 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext3)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should store data
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data","total":44,"completed":17,"skipped":961,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
Nov 30 03:01:57.604: INFO: Running AfterSuite actions on all nodes
Nov 30 03:01:57.604: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 30 03:01:57.604: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 200 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow]
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should access to two volumes with the same volume mode and retain data across pod recreation on different node
    [90mtest/e2e/storage/testsuites/multivolume.go:168[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":36,"completed":14,"skipped":846,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] volumeMode[0m 
  [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m
  [37mtest/e2e/storage/testsuites/volumemode.go:299[0m

[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 30 03:01:27.105: INFO: >>> kubeConfig: /root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename volumemode
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  test/e2e/storage/testsuites/volumemode.go:299
Nov 30 03:01:27.864: INFO: Creating resource for dynamic PV
Nov 30 03:01:27.864: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} 
[1mSTEP[0m: creating a StorageClass volumemode-673-e2e-sczrtn8
[1mSTEP[0m: creating a claim
Nov 30 03:01:28.086: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com7phbh] to have phase Bound
Nov 30 03:01:28.195: INFO: PersistentVolumeClaim test.csi.azure.com7phbh found but phase is Pending instead of Bound.
Nov 30 03:01:30.305: INFO: PersistentVolumeClaim test.csi.azure.com7phbh found but phase is Pending instead of Bound.
Nov 30 03:01:32.415: INFO: PersistentVolumeClaim test.csi.azure.com7phbh found and phase=Bound (4.328591103s)
[1mSTEP[0m: Creating pod
[1mSTEP[0m: Waiting for the pod to fail
Nov 30 03:01:32.962: INFO: Deleting pod "pod-fcd390fd-0d3e-4d4f-a31e-8fe9056ad787" in namespace "volumemode-673"
Nov 30 03:01:33.073: INFO: Wait up to 5m0s for pod "pod-fcd390fd-0d3e-4d4f-a31e-8fe9056ad787" to be fully deleted
[1mSTEP[0m: Deleting pvc
Nov 30 03:01:35.292: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com7phbh"
Nov 30 03:01:35.403: INFO: Waiting up to 5m0s for PersistentVolume pvc-6e12467f-3c94-4766-89c8-1f0fef8bf1fb to get deleted
Nov 30 03:01:35.511: INFO: PersistentVolume pvc-6e12467f-3c94-4766-89c8-1f0fef8bf1fb found and phase=Released (108.198003ms)
... skipping 32 lines ...

[32m• [SLOW TEST:141.820 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should fail to use a volume in a pod with mismatched mode [Slow]
    [90mtest/e2e/storage/testsuites/volumemode.go:299[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":30,"completed":12,"skipped":1013,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] volumes[0m 
  [1mshould store data[0m
  [37mtest/e2e/storage/testsuites/volumes.go:161[0m
... skipping 108 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (ext4)] volumes
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should store data
    [90mtest/e2e/storage/testsuites/volumes.go:161[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data","total":31,"completed":12,"skipped":929,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral[0m 
  [1mshould create read/write inline ephemeral volume[0m
  [37mtest/e2e/storage/testsuites/ephemeral.go:196[0m
... skipping 64 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should create read/write inline ephemeral volume
    [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":30,"completed":13,"skipped":1019,"failed":0}
Nov 30 03:08:03.972: INFO: Running AfterSuite actions on all nodes
Nov 30 03:08:03.972: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 30 03:08:03.972: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
Nov 30 03:08:03.972: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func8.2
Nov 30 03:08:03.972: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage/vsphere.glob..func17.3
Nov 30 03:08:03.972: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage/vsphere.glob..func9.2
... skipping 134 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (default fs)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    [90mtest/e2e/storage/testsuites/provisioning.go:208[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":36,"completed":15,"skipped":864,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Inline-volume (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 03:08:19.245: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping
... skipping 38 lines ...
Nov 30 03:03:50.093: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comhnqjj] to have phase Bound
Nov 30 03:03:50.207: INFO: PersistentVolumeClaim test.csi.azure.comhnqjj found but phase is Pending instead of Bound.
Nov 30 03:03:52.317: INFO: PersistentVolumeClaim test.csi.azure.comhnqjj found but phase is Pending instead of Bound.
Nov 30 03:03:54.426: INFO: PersistentVolumeClaim test.csi.azure.comhnqjj found and phase=Bound (4.332945095s)
[1mSTEP[0m: [init] starting a pod to use the claim
[1mSTEP[0m: [init] check pod success
Nov 30 03:03:54.877: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-56s5x" in namespace "snapshotting-5743" to be "Succeeded or Failed"
Nov 30 03:03:54.987: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 110.022536ms
Nov 30 03:03:57.098: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 2.22116241s
Nov 30 03:03:59.209: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 4.331637125s
Nov 30 03:04:01.319: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 6.442037622s
Nov 30 03:04:03.431: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 8.553682082s
Nov 30 03:04:05.542: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 10.665508908s
... skipping 29 lines ...
Nov 30 03:05:08.880: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m14.00305922s
Nov 30 03:05:10.991: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.114181817s
Nov 30 03:05:13.102: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.224950239s
Nov 30 03:05:15.213: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.335898184s
Nov 30 03:05:17.323: INFO: Pod "pvc-snapshottable-tester-56s5x": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m22.446486984s
[1mSTEP[0m: Saw pod success
Nov 30 03:05:17.324: INFO: Pod "pvc-snapshottable-tester-56s5x" satisfied condition "Succeeded or Failed"
[1mSTEP[0m: [init] checking the claim
Nov 30 03:05:17.434: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comhnqjj] to have phase Bound
Nov 30 03:05:17.546: INFO: PersistentVolumeClaim test.csi.azure.comhnqjj found and phase=Bound (112.038809ms)
[1mSTEP[0m: [init] checking the PV
[1mSTEP[0m: [init] deleting the pod
Nov 30 03:05:17.878: INFO: Pod pvc-snapshottable-tester-56s5x has the following logs: 
... skipping 15 lines ...
Nov 30 03:05:29.761: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-75bee6f1-f004-44f4-9960-03e6090ce8bf creationTime:2022-11-30T03:05:25Z readyToUse:true restoreSize:5Gi]
Nov 30 03:05:29.761: INFO: snapshotContentName snapcontent-75bee6f1-f004-44f4-9960-03e6090ce8bf
[1mSTEP[0m: checking the snapshot
[1mSTEP[0m: checking the SnapshotContent
[1mSTEP[0m: Modifying source data test
[1mSTEP[0m: modifying the data in the source PVC
Nov 30 03:05:30.209: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-qvbq7" in namespace "snapshotting-5743" to be "Succeeded or Failed"
Nov 30 03:05:30.319: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 109.784445ms
Nov 30 03:05:32.429: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 2.220080549s
Nov 30 03:05:34.539: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 4.330122349s
Nov 30 03:05:36.651: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441889592s
Nov 30 03:05:38.763: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 8.55438s
Nov 30 03:05:40.874: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 10.664987379s
... skipping 32 lines ...
Nov 30 03:06:50.556: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.347391173s
Nov 30 03:06:52.667: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.457833013s
Nov 30 03:06:54.783: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.573830049s
Nov 30 03:06:56.893: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.684164215s
Nov 30 03:06:59.004: INFO: Pod "pvc-snapshottable-data-tester-qvbq7": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m28.79502884s
[1mSTEP[0m: Saw pod success
Nov 30 03:06:59.004: INFO: Pod "pvc-snapshottable-data-tester-qvbq7" satisfied condition "Succeeded or Failed"
Nov 30 03:06:59.255: INFO: Pod pvc-snapshottable-data-tester-qvbq7 has the following logs: 
Nov 30 03:06:59.255: INFO: Deleting pod "pvc-snapshottable-data-tester-qvbq7" in namespace "snapshotting-5743"
Nov 30 03:06:59.371: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-qvbq7" to be fully deleted
[1mSTEP[0m: creating a pvc from the snapshot
[1mSTEP[0m: starting a pod to use the snapshot
Nov 30 03:07:51.929: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-5bbt3k3k.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-5743 exec restored-pvc-tester-xcpts --namespace=snapshotting-5743 -- cat /mnt/test/data'
... skipping 32 lines ...
Nov 30 03:08:17.163: INFO: volumesnapshotcontents snapcontent-75bee6f1-f004-44f4-9960-03e6090ce8bf has been found and is not deleted
Nov 30 03:08:18.274: INFO: volumesnapshotcontents snapcontent-75bee6f1-f004-44f4-9960-03e6090ce8bf has been found and is not deleted
Nov 30 03:08:19.385: INFO: volumesnapshotcontents snapcontent-75bee6f1-f004-44f4-9960-03e6090ce8bf has been found and is not deleted
Nov 30 03:08:20.496: INFO: volumesnapshotcontents snapcontent-75bee6f1-f004-44f4-9960-03e6090ce8bf has been found and is not deleted
Nov 30 03:08:21.608: INFO: volumesnapshotcontents snapcontent-75bee6f1-f004-44f4-9960-03e6090ce8bf has been found and is not deleted
Nov 30 03:08:22.718: INFO: volumesnapshotcontents snapcontent-75bee6f1-f004-44f4-9960-03e6090ce8bf has been found and is not deleted
Nov 30 03:08:23.718: INFO: WaitUntil failed after reaching the timeout 30s
[AfterEach] volume snapshot controller
  test/e2e/storage/testsuites/snapshottable.go:172
Nov 30 03:08:23.835: INFO: Pod restored-pvc-tester-xcpts has the following logs: 
Nov 30 03:08:23.835: INFO: Deleting pod "restored-pvc-tester-xcpts" in namespace "snapshotting-5743"
Nov 30 03:08:23.945: INFO: Wait up to 5m0s for pod "restored-pvc-tester-xcpts" to be fully deleted
Nov 30 03:08:26.166: INFO: deleting claim "snapshotting-5743"/"pvc-vszrm"
... skipping 31 lines ...
    [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m
      
      [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m
        should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)
        [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":31,"completed":13,"skipped":939,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
Nov 30 03:08:44.467: INFO: Running AfterSuite actions on all nodes
Nov 30 03:08:44.467: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2
Nov 30 03:08:44.467: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2
... skipping 132 lines ...
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (block volmode)] provisioning
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]
    [90mtest/e2e/storage/testsuites/provisioning.go:208[0m
[90m------------------------------[0m
{"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":36,"completed":16,"skipped":901,"failed":0}

[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m
[90m------------------------------[0m
[BeforeEach] [Testpattern: Pre-provisioned PV (ext3)] volumes
  test/e2e/storage/framework/testsuite.go:51
Nov 30 03:10:52.560: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 53 lines ...
  test/e2e/framework/framework.go:187
[1mSTEP[0m: Creating a kubernetes client
Nov 30 03:10:53.597: INFO: >>> kubeConfig: /root/tmp1439426005/kubeconfig/kubeconfig.westeurope.json
[1mSTEP[0m: Building a namespace api object, basename topology
[1mSTEP[0m: Waiting for a default service account to be provisioned in namespace
[1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  test/e2e/storage/testsuites/topology.go:194
Nov 30 03:10:54.356: INFO: Driver didn't provide topology keys -- skipping
[AfterEach] [Testpattern: Dynamic PV (immediate binding)] topology
  test/e2e/framework/framework.go:188
Nov 30 03:10:54.357: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
[1mSTEP[0m: Destroying namespace "topology-3578" for this suite.


[36m[1mS [SKIPPING] [0.981 seconds][0m
External Storage [Driver: test.csi.azure.com]
[90mtest/e2e/storage/external/external.go:174[0m
  [Testpattern: Dynamic PV (immediate binding)] topology
  [90mtest/e2e/storage/framework/testsuite.go:50[0m
    [36m[1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement][0m
    [90mtest/e2e/storage/testsuites/topology.go:194[0m

    [36mDriver didn't provide topology keys -- skipping[0m

    test/e2e/storage/testsuites/topology.go:126
[90m------------------------------[0m
... skipping 20 lines ...
Nov 30 03:10:54.649: INFO: Running AfterSuite actions on node 1



[91m[1mSummarizing 1 Failure:[0m

[91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (default fs)] subPath [0m[91m[1m[Measurement] should support non-existent path [0m
[37mtest/e2e/storage/testsuites/subpath.go:185[0m

[1m[91mRan 89 of 7227 Specs in 3497.327 seconds[0m
[1m[91mFAIL![0m -- [32m[1m88 Passed[0m | [91m[1m1 Failed[0m | [33m[1m0 Pending[0m | [36m[1m7138 Skipped[0m 

Ginkgo ran 1 suite in 58m20.92314062s
Test Suite Failed
+ print_logs
+ sed -i s/disk.csi.azure.com/test.csi.azure.com/g deploy/example/storageclass-azuredisk-csi.yaml
+ bash ./hack/verify-examples.sh linux azurepubliccloud ephemeral test
begin to create deployment examples ...
storageclass.storage.k8s.io/managed-csi created
Applying config "deploy/example/deployment.yaml"
... skipping 100 lines ...
I1130 02:12:28.901082       1 reflector.go:257] Listing and watching *v1beta2.AzVolume from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117
I1130 02:12:28.901432       1 reflector.go:221] Starting reflector *v1beta2.AzVolumeAttachment (30s) from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117
I1130 02:12:28.901453       1 reflector.go:257] Listing and watching *v1beta2.AzVolumeAttachment from sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117
I1130 02:12:29.000892       1 shared_informer.go:285] caches populated
I1130 02:12:29.000951       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb e2e-test
I1130 02:12:29.000964       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1130 02:12:29.004222       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1130 02:12:29.004247       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1130 02:12:29.004254       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1130 02:12:29.004283       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1130 02:12:29.005067       1 azure_auth.go:245] Using AzurePublicCloud environment
I1130 02:12:29.005088       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1130 02:12:29.005094       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 29 lines ...
I1130 02:12:29.005570       1 azure_vmasclient.go:73] Azure AvailabilitySetsClient  (write ops) using rate limit config: QPS=100, bucket=1000
I1130 02:12:29.005632       1 azure.go:1039] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333000, Bucket: 240
I1130 02:12:29.005686       1 azuredisk_v2.go:215] disable UseInstanceMetadata for controller
I1130 02:12:29.005698       1 azuredisk_v2.go:225] DisableAvailabilitySetNodes for controller since current VMType is vmss
I1130 02:12:29.005707       1 azuredisk_v2.go:231] cloud: AzurePublicCloud, location: westeurope, rg: kubetest-5bbt3k3k, VMType: vmss, PrimaryScaleSetName: k8s-agentpool1-10232909-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: true
I1130 02:12:29.005715       1 skus.go:123] NewNodeInfo: Starting to populate node and disk sku information.
E1130 02:12:29.005730       1 azuredisk_v2.go:239] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance
I1130 02:12:29.005842       1 mount_linux.go:208] Detected OS without systemd
I1130 02:12:29.005869       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I1130 02:12:29.005876       1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I1130 02:12:29.005881       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I1130 02:12:29.005886       1 driver.go:81] Enabling controller service capability: CLONE_VOLUME
I1130 02:12:29.005892       1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
... skipping 338 lines ...
I1130 02:12:29.335031       1 shared_state.go:485]  "msg"="Storing pod azure-ip-masq-agent-hsdcx and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa"
I1130 02:12:29.335048       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/azure-ip-masq-agent-hsdcx." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-hsdcx"
I1130 02:12:29.335069       1 shared_state.go:309]  "msg"="Getting requested volumes for pod (kube-system/azure-ip-masq-agent-hsdcx)." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-hsdcx"
I1130 02:12:29.335084       1 pod.go:99]  "msg"="Pod kube-system/azure-ip-masq-agent-hsdcx has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-hsdcx"
I1130 02:12:29.335099       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/azure-ip-masq-agent-hsdcx" "latency"=52204
I1130 02:12:29.335111       1 shared_state.go:415]  "msg"="Adding pod csi-test-controller-854d9ff965-cdbxk to shared map with keyName kube-system/csi-test-controller-854d9ff965-cdbxk." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa"
I1130 02:12:29.335455       1 shared_state.go:421]  "msg"="Pod spec of pod csi-test-controller-854d9ff965-cdbxk is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-vmnr7 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-vmnr7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-vmnr7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-vmnr7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-vmnr7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-vmnr7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-vmnr7 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000aa28c8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-10232909-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000aa28d0} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000aa28d8}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000aa28e0 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000aa28e4 PreemptionPolicy:0xc00007e600 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil>}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-vmnr7 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa"
I1130 02:12:29.335498       1 shared_state.go:459]  "msg"="Pod csi-test-controller-854d9ff965-cdbxk: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa"
I1130 02:12:29.335519       1 shared_state.go:459]  "msg"="Pod csi-test-controller-854d9ff965-cdbxk: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa"
I1130 02:12:29.335556       1 shared_state.go:459]  "msg"="Pod csi-test-controller-854d9ff965-cdbxk: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa"
I1130 02:12:29.335590       1 shared_state.go:459]  "msg"="Pod csi-test-controller-854d9ff965-cdbxk: Skipping Volume {kube-api-access-vmnr7 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa"
I1130 02:12:29.335602       1 shared_state.go:485]  "msg"="Storing pod csi-test-controller-854d9ff965-cdbxk and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa"
I1130 02:12:29.335625       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-test-controller-854d9ff965-cdbxk." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-854d9ff965-cdbxk"
... skipping 104 lines ...
I1130 02:12:29.340346       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-test-node-hjgjk." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi/azure.com/pod-name"="kube-system/csi-test-node-hjgjk"
I1130 02:12:29.340367       1 shared_state.go:309]  "msg"="Getting requested volumes for pod (kube-system/csi-test-node-hjgjk)." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi/azure.com/pod-name"="kube-system/csi-test-node-hjgjk"
I1130 02:12:29.340379       1 pod.go:99]  "msg"="Pod kube-system/csi-test-node-hjgjk has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi/azure.com/pod-name"="kube-system/csi-test-node-hjgjk"
I1130 02:12:29.340398       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/csi-test-node-hjgjk" "latency"=56403
I1130 02:12:29.340415       1 pod.go:150]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).Recover" "latency"=15774722
I1130 02:12:29.378624       1 azuredisk_v2.go:420]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="70db8b79-7054-11ed-803b-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).StartControllersAndDieOnExit.func1" "latency"=243264727
I1130 02:12:29.395768       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-master-10232909-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I1130 02:12:29.395824       1 shared_state.go:1066]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="7103492d-7054-11ed-803b-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8300
I1130 02:12:29.395886       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I1130 02:12:29.395901       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I1130 02:12:29.395954       1 shared_state.go:1066]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="71034da9-7054-11ed-803b-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8601
I1130 02:12:29.395957       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I1130 02:12:29.396001       1 shared_state.go:1066]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="71035078-7054-11ed-803b-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=6000
I1130 02:12:29.438307       1 shared_state.go:415]  "msg"="Adding pod kube-addon-manager-k8s-master-10232909-0 to shared map with keyName kube-system/kube-addon-manager-k8s-master-10232909-0." 
I1130 02:12:29.438588       1 shared_state.go:421]  "msg"="Pod spec of pod kube-addon-manager-k8s-master-10232909-0 is: {Volumes:[{Name:addons VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/addons,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:msi VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/waagent/ManagedIdentity-Settings,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-lib-kubelet VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/kubelet,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:etc-kubernetes VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:kube-addon-manager Image:mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5 Command:[] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:KUBECONFIG Value:/var/lib/kubelet/kubeconfig ValueFrom:nil} {Name:ADDON_PATH Value:/etc/kubernetes/addons ValueFrom:nil}] Resources:{Limits:map[] Requests:map[cpu:{i:{value:5 scale:-3} d:{Dec:<nil>} s:5m Format:DecimalSI} memory:{i:{value:52428800 scale:0} d:{Dec:<nil>} s:50Mi Format:BinarySI}]} VolumeMounts:[{Name:addons ReadOnly:true MountPath:/etc/kubernetes/addons SubPath: MountPropagation:<nil> SubPathExpr:} {Name:msi ReadOnly:true MountPath:/var/lib/waagent/ManagedIdentity-Settings SubPath: MountPropagation:<nil> SubPathExpr:} {Name:var-lib-kubelet ReadOnly:true MountPath:/var/lib/kubelet SubPath: MountPropagation:<nil> SubPathExpr:} {Name:etc-kubernetes ReadOnly:true MountPath:/etc/kubernetes SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000df4a50 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken:<nil> NodeName:k8s-master-10232909-0 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key: Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>}] HostAliases:[] PriorityClassName:system-node-critical Priority:0xc000df4a58 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000df4a5c PreemptionPolicy:0xc000c254c0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil>}. With volumes: [{Name:addons VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/addons,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:msi VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/waagent/ManagedIdentity-Settings,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-lib-kubelet VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/kubelet,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:etc-kubernetes VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I1130 02:12:29.438639       1 shared_state.go:459]  "msg"="Pod kube-addon-manager-k8s-master-10232909-0: Skipping Volume {addons {&HostPathVolumeSource{Path:/etc/kubernetes/addons,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I1130 02:12:29.438667       1 shared_state.go:459]  "msg"="Pod kube-addon-manager-k8s-master-10232909-0: Skipping Volume {msi {&HostPathVolumeSource{Path:/var/lib/waagent/ManagedIdentity-Settings,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I1130 02:12:29.438694       1 shared_state.go:459]  "msg"="Pod kube-addon-manager-k8s-master-10232909-0: Skipping Volume {var-lib-kubelet {&HostPathVolumeSource{Path:/var/lib/kubelet,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
... skipping 32946 lines ...
I1130 02:56:05.097522       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-42b83320-2e34-4e17-a558-43ec0e08855d/globalmount
I1130 02:56:05.097555       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-42b83320-2e34-4e17-a558-43ec0e08855d/globalmount" is a mountpoint, unmounting
I1130 02:56:05.097564       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-42b83320-2e34-4e17-a558-43ec0e08855d/globalmount
W1130 02:56:05.119282       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-42b83320-2e34-4e17-a558-43ec0e08855d/globalmount" is not a mountpoint, deleting
I1130 02:56:05.119406       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-42b83320-2e34-4e17-a558-43ec0e08855d/globalmount successfully
I1130 02:56:05.119423       1 utils.go:85] GRPC response: {}
I1130 02:56:18.655097       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:56:18.655239       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="865ad454-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3d809ca8-c4a7-4ec3-aff0-c60c27a37363" "latency"=16473042283
I1130 02:56:18.655302       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="865ad454-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16473238481
I1130 02:56:19.328278       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:56:19Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="90931f8c-705a-11ed-ab94-000d3aad6f3e"
I1130 02:56:19.336912       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 8 milliseconds
I1130 02:56:19.337125       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="90931f8c-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8903816
I1130 02:56:19.397390       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
... skipping 39 lines ...
I1130 02:56:19.830029       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:56:19.830049       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9","csi.storage.k8s.io/pvc/name":"test.csi.azure.comlxdzz","csi.storage.k8s.io/pvc/namespace":"multivolume-1878","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9"}
I1130 02:56:19.830270       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9-k8s-agentpool1-10232909-vmss000001-attachment)
I1130 02:56:20.661100       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:56:20.661122       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:56:20.661190       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:56:20.661491       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:56:20.787489       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:56:26.463394       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1130 02:56:26.463422       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/7d0fb956-4cf2-4138-b4e2-d4f88e1e9864/volumes/kubernetes.io~csi/pvc-3d809ca8-c4a7-4ec3-aff0-c60c27a37363/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-3d809ca8-c4a7-4ec3-aff0-c60c27a37363"}
I1130 02:56:26.463580       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-3d809ca8-c4a7-4ec3-aff0-c60c27a37363 on /var/lib/kubelet/pods/7d0fb956-4cf2-4138-b4e2-d4f88e1e9864/volumes/kubernetes.io~csi/pvc-3d809ca8-c4a7-4ec3-aff0-c60c27a37363/mount
I1130 02:56:26.463614       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/7d0fb956-4cf2-4138-b4e2-d4f88e1e9864/volumes/kubernetes.io~csi/pvc-3d809ca8-c4a7-4ec3-aff0-c60c27a37363/mount" is a mountpoint, unmounting
I1130 02:56:26.463636       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/7d0fb956-4cf2-4138-b4e2-d4f88e1e9864/volumes/kubernetes.io~csi/pvc-3d809ca8-c4a7-4ec3-aff0-c60c27a37363/mount
... skipping 38 lines ...
I1130 02:56:49.334297       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:56:49Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="a275ae88-705a-11ed-ab94-000d3aad6f3e"
I1130 02:56:49.348459       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 13 milliseconds
I1130 02:56:49.348784       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="a275ae88-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14505166
I1130 02:56:50.661722       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:56:50.661767       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:56:50.661750       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:56:50.661900       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:56:50.788285       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:57:03.826677       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:57:03.826705       1 utils.go:79] GRPC request: {}
I1130 02:57:03.826775       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:57:19.058825       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:57:19.058917       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="90dfba9a-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9" "latency"=59228562152
I1130 02:57:19.058943       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="90dfba9a-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59228694551
I1130 02:57:19.329170       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:57:19Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="b45688af-705a-11ed-ab94-000d3aad6f3e"
I1130 02:57:19.337289       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 7 milliseconds
I1130 02:57:19.337467       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="b45688af-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8343622
I1130 02:57:19.818587       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
... skipping 115 lines ...
I1130 02:58:03.825809       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:58:19.328756       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:58:19Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="d819be40-705a-11ed-ab94-000d3aad6f3e"
I1130 02:58:19.336665       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 7 milliseconds
I1130 02:58:19.336899       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="d819be40-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8197525
I1130 02:58:20.664278       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:58:20.664331       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:58:20.664486       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:58:20.664532       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:58:20.664271       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:58:20.790767       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:58:33.826978       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:58:33.827009       1 utils.go:79] GRPC request: {}
I1130 02:58:33.827069       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:58:49.333824       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:58:49Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="e9fc2801-705a-11ed-ab94-000d3aad6f3e"
I1130 02:58:49.348463       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 14 milliseconds
I1130 02:58:49.348792       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="e9fc2801-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14988676
I1130 02:58:50.665258       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:58:50.665293       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:58:50.665366       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:58:50.665410       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:58:50.665434       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:58:50.791946       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:59:03.825740       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:59:03.825767       1 utils.go:79] GRPC request: {}
I1130 02:59:03.825817       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:59:06.270009       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:59:06.270117       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="c9ecdf97-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7d97a3fc-bf2a-4c9a-8f44-0df0a902998a" "latency"=70723473442
I1130 02:59:06.270174       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c9ecdf97-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=70723642941
I1130 02:59:06.276132       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:59:06.276229       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="c99fa38e-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1f66396d-3217-4913-94bb-2d0a1199e88b" "latency"=71235741977
I1130 02:59:06.276282       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c99fa38e-705a-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=71235910576
I1130 02:59:07.047712       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdd under /dev/disk/azure/scsi1/
I1130 02:59:07.047767       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
I1130 02:59:07.047807       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7d97a3fc-bf2a-4c9a-8f44-0df0a902998a/globalmount with mount options([])
I1130 02:59:07.047821       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun4" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun4])
... skipping 126 lines ...
I1130 02:59:57.854880       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:59:57.854900       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d155bb35-b80d-47e0-8263-6f07abe1c3ab/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d155bb35-b80d-47e0-8263-6f07abe1c3ab","csi.storage.k8s.io/pvc/name":"test.csi.azure.com9jmlw","csi.storage.k8s.io/pvc/namespace":"multivolume-7878","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-d155bb35-b80d-47e0-8263-6f07abe1c3ab"}
I1130 02:59:57.855105       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-d155bb35-b80d-47e0-8263-6f07abe1c3ab-k8s-agentpool1-10232909-vmss000001-attachment)
I1130 03:00:03.827244       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:00:03.827272       1 utils.go:79] GRPC request: {}
I1130 03:00:03.827334       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:00:12.407758       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:00:12.407857       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="12d3adc9-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d155bb35-b80d-47e0-8263-6f07abe1c3ab" "latency"=14552646148
I1130 03:00:12.407915       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="12d3adc9-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14552799246
I1130 03:00:14.301223       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 03:00:14.301278       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 03:00:14.301310       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d155bb35-b80d-47e0-8263-6f07abe1c3ab/globalmount with mount options([nouuid])
I1130 03:00:14.301335       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 200 lines ...
I1130 03:02:19.332100       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:02:19Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="67275862-705b-11ed-ab94-000d3aad6f3e"
I1130 03:02:19.341913       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 9 milliseconds
I1130 03:02:19.342113       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="67275862-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10077833
I1130 03:02:20.670593       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:02:20.670635       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:02:20.670683       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:02:20.670741       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:02:20.670766       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:02:20.796105       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:02:27.422507       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:02:27.422619       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="5a1ce304-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b7a9bbf5-547f-432b-870f-cc595e3cb25a" "latency"=29969411112
I1130 03:02:27.422664       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5a1ce304-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29969551412
I1130 03:02:27.426730       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:02:27.426818       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="5a01b4e4-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e4897057-0b50-43ce-893a-455eba31ac78" "latency"=30151661507
I1130 03:02:27.426855       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5a01b4e4-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30151887906
I1130 03:02:28.193281       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1130 03:02:28.193331       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 03:02:28.193372       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e4897057-0b50-43ce-893a-455eba31ac78/globalmount with mount options([nouuid])
I1130 03:02:28.193388       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 154 lines ...
I1130 03:04:03.826514       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:04:19.333832       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:04:19Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="aeae289b-705b-11ed-ab94-000d3aad6f3e"
I1130 03:04:19.342093       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 7 milliseconds
I1130 03:04:19.342798       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="aeae289b-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9017126
I1130 03:04:20.672303       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:20.672405       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:20.672446       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:04:20.672465       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:20.672472       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:04:20.797839       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:33.826408       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:04:33.826435       1 utils.go:79] GRPC request: {}
I1130 03:04:33.826501       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:04:49.333596       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:04:49Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="c08fc279-705b-11ed-ab94-000d3aad6f3e"
I1130 03:04:49.342553       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 8 milliseconds
I1130 03:04:49.342766       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="c08fc279-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9216923
I1130 03:04:50.673248       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:50.673318       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:50.673328       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:50.673428       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:04:50.673475       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:04:50.798701       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:05:00.622514       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 101 items received
I1130 03:05:00.628849       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=28464&timeout=6m55s&timeoutSeconds=415&watch=true 200 OK in 6 milliseconds
I1130 03:05:03.826307       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:05:03.826343       1 utils.go:79] GRPC request: {}
I1130 03:05:03.826400       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:05:11.428992       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:05:11.429103       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="a46c2b8b-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9bd5a444-65bf-4630-a882-2da53aadb4bf" "latency"=69304863182
I1130 03:05:11.429130       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a46c2b8b-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=69305045680
I1130 03:05:11.432059       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:05:11.432128       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="a46c3931-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b36ef2a0-b424-4e4a-ac62-ba762a82768a" "latency"=69307587863
I1130 03:05:11.432153       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a46c3931-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=69307719362
I1130 03:05:12.194220       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 03:05:12.194281       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 03:05:12.194347       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9bd5a444-65bf-4630-a882-2da53aadb4bf/globalmount with mount options([])
I1130 03:05:12.194369       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 126 lines ...
I1130 03:05:49.328527       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:05:49Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="e4524210-705b-11ed-ab94-000d3aad6f3e"
I1130 03:05:49.337398       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 8 milliseconds
I1130 03:05:49.337715       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="e4524210-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9224434
I1130 03:05:50.674160       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:05:50.674251       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:05:50.674304       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:05:50.674321       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:05:50.799538       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:03.827627       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:06:03.827660       1 utils.go:79] GRPC request: {}
I1130 03:06:03.827719       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:06:19.334220       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:06:19Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="f634c48a-705b-11ed-ab94-000d3aad6f3e"
I1130 03:06:19.351367       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 16 milliseconds
I1130 03:06:19.351628       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="f634c48a-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=17428541
I1130 03:06:20.675134       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:20.675162       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:20.675195       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:20.675239       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:06:20.800637       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:33.826340       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:06:33.826375       1 utils.go:79] GRPC request: {}
I1130 03:06:33.826438       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:06:49.333475       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:06:49Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="08164a49-705c-11ed-ab94-000d3aad6f3e"
I1130 03:06:49.350027       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 16 milliseconds
I1130 03:06:49.350344       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="08164a49-705c-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16896267
I1130 03:06:50.676164       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:50.676207       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:50.676266       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:50.676275       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:06:50.754751       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:06:50.754857       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="e137aa80-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7829b989-770c-4d49-b256-e7db99b0c4ec" "latency"=66633723736
I1130 03:06:50.754912       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e137aa80-705b-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66633895334
I1130 03:06:50.801643       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:51.537030       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1130 03:06:51.537088       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 03:06:51.537132       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7829b989-770c-4d49-b256-e7db99b0c4ec/globalmount with mount options([])
... skipping 154 lines ...
I1130 03:11:30.291297       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:11:30.291320       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-748294a3-d12e-45bd-91aa-9df2f0c04871/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-748294a3-d12e-45bd-91aa-9df2f0c04871","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-748294a3-d12e-45bd-91aa-9df2f0c04871"}
I1130 03:11:30.291587       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-748294a3-d12e-45bd-91aa-9df2f0c04871-k8s-agentpool1-10232909-vmss000001-attachment)
I1130 03:11:33.826143       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:11:33.826173       1 utils.go:79] GRPC request: {}
I1130 03:11:33.826230       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:11:37.523014       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:11:37.523141       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="af8d1922-705c-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-748294a3-d12e-45bd-91aa-9df2f0c04871" "latency"=7231426256
I1130 03:11:37.523183       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="af8d1922-705c-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7231625553
I1130 03:11:39.362869       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 03:11:39.362926       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1130 03:11:39.362961       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-748294a3-d12e-45bd-91aa-9df2f0c04871/globalmount with mount options([])
I1130 03:11:39.362972       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 59 lines ...
I1130 03:12:21.251548       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 03:12:21.251575       1 utils.go:79] GRPC request: {}
I1130 03:12:21.251615       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 03:12:21.252561       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:12:21.252580       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-36ea6528-0dc0-4377-a8fa-1f6686f8b178/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-36ea6528-0dc0-4377-a8fa-1f6686f8b178","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-36ea6528-0dc0-4377-a8fa-1f6686f8b178"}
I1130 03:12:21.252816       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-36ea6528-0dc0-4377-a8fa-1f6686f8b178-k8s-agentpool1-10232909-vmss000001-attachment)
I1130 03:12:30.450317       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:12:30.450407       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="cded2ad2-705c-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-36ea6528-0dc0-4377-a8fa-1f6686f8b178" "latency"=9197509816
I1130 03:12:30.450442       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cded2ad2-705c-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9197640114
I1130 03:12:31.195627       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1130 03:12:31.195680       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I1130 03:12:31.195725       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-36ea6528-0dc0-4377-a8fa-1f6686f8b178/globalmount with mount options([])
I1130 03:12:31.195747       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 43 lines ...
I1130 03:12:49.333903       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:12:49Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="dea9ff27-705c-11ed-ab94-000d3aad6f3e"
I1130 03:12:49.349347       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000001/status 200 OK in 15 milliseconds
I1130 03:12:49.349746       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="dea9ff27-705c-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15872550
I1130 03:12:50.686019       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:12:50.686089       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:12:50.686030       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:12:50.686453       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:12:50.810406       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:12:57.134448       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:12:57.134555       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000001" "disk.csi.azure.com/request-id"="da3d3faf-705c-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8c8e3101-40e9-40ef-96d8-d1553ea99188" "latency"=15224155734
I1130 03:12:57.134586       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="da3d3faf-705c-11ed-ab94-000d3aad6f3e" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15224307832
I1130 03:12:57.997456       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1130 03:12:57.997526       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType StandardSSD_LRS
I1130 03:12:57.997587       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8c8e3101-40e9-40ef-96d8-d1553ea99188/globalmount with mount options([])
I1130 03:12:57.997598       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 141 lines ...
I1130 02:12:27.462720       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=2585&timeout=7m36s&timeoutSeconds=456&watch=true 200 OK in 1 milliseconds
I1130 02:12:27.463255       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=2536&timeout=6m57s&timeoutSeconds=417&watch=true 200 OK in 1 milliseconds
I1130 02:12:27.558564       1 shared_informer.go:285] caches populated
I1130 02:12:27.558621       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb e2e-test
I1130 02:12:27.558668       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1130 02:12:27.562547       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds
I1130 02:12:27.562879       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1130 02:12:27.562903       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1130 02:12:27.562916       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1130 02:12:27.562935       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1130 02:12:27.564097       1 azure_auth.go:245] Using AzurePublicCloud environment
I1130 02:12:27.564120       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1130 02:12:27.564125       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 53 lines ...
I1130 02:12:27.697675       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes 201 Created in 5 milliseconds
I1130 02:12:27.702332       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 4 milliseconds
I1130 02:12:27.702583       1 azuredisk_v2.go:532]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="6fff14ef-7054-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).registerAzDriverNode" "latency"=11997909
I1130 02:12:27.702694       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:12:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="7000eee1-7054-11ed-9767-000d3aad6ffc"
I1130 02:12:27.702998       1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I1130 02:12:27.705825       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 409 Conflict in 3 milliseconds
E1130 02:12:27.705972       1 azuredisk_v2.go:591]  "msg"="Failed to update AzDriverNode status after creation" "error"="Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-10232909-vmss000002\": the object has been modified; please apply your changes to the latest version and try again" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="7000eee1-7054-11ed-9767-000d3aad6ffc"
E1130 02:12:27.706045       1 azuredisk_v2.go:543]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-10232909-vmss000002\": the object has been modified; please apply your changes to the latest version and try again]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="7000eee1-7054-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=3343530
I1130 02:12:27.706057       1 azuredisk_v2.go:506] Starting heartbeat loop with initial delay 15.236s and frequency 30s
I1130 02:12:28.554130       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1130 02:12:28.554215       1 utils.go:79] GRPC request: {}
I1130 02:12:28.555923       1 utils.go:85] GRPC response: {"name":"test.csi.azure.com","vendor_version":"latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb"}
I1130 02:12:28.629999       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1130 02:12:28.630027       1 utils.go:79] GRPC request: {}
... skipping 16 lines ...
I1130 02:12:55.129135       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:12:55.129158       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-02e1f3b6-adc8-49b9-9dfe-173edf576124/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-02e1f3b6-adc8-49b9-9dfe-173edf576124","csi.storage.k8s.io/pvc/name":"test.csi.azure.comrjt4l","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-292","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-02e1f3b6-adc8-49b9-9dfe-173edf576124"}
I1130 02:12:55.129913       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-02e1f3b6-adc8-49b9-9dfe-173edf576124-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:12:57.461801       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:12:57.461835       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:12:57.462989       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:12:57.463089       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:12:57.602437       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:13:03.733199       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:13:03.733228       1 utils.go:79] GRPC request: {}
I1130 02:13:03.733310       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:13:12.945034       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:13:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="8af85be8-7054-11ed-9767-000d3aad6ffc"
I1130 02:13:12.955164       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 9 milliseconds
I1130 02:13:12.955681       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="8af85be8-7054-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10710195
I1130 02:13:27.084523       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:13:27.084602       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="8059f9f3-7054-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-02e1f3b6-adc8-49b9-9dfe-173edf576124" "latency"=31954571904
I1130 02:13:27.084627       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8059f9f3-7054-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31954862308
I1130 02:13:27.462799       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:13:27.462932       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:13:27.463100       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:13:27.602885       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 161 lines ...
I1130 02:15:28.991442       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:15:28.991480       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5f1d3684-1385-48b2-bf91-38e7276a92ef/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5f1d3684-1385-48b2-bf91-38e7276a92ef","csi.storage.k8s.io/pvc/name":"test.csi.azure.comtx7tl","csi.storage.k8s.io/pvc/namespace":"provisioning-5508","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-5f1d3684-1385-48b2-bf91-38e7276a92ef"}
I1130 02:15:28.991762       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5f1d3684-1385-48b2-bf91-38e7276a92ef-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:15:33.733077       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:15:33.733098       1 utils.go:79] GRPC request: {}
I1130 02:15:33.733144       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:15:34.573642       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:15:34.573736       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="dc0f7342-7054-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5f1d3684-1385-48b2-bf91-38e7276a92ef" "latency"=5581881134
I1130 02:15:34.573791       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="dc0f7342-7054-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5582047936
I1130 02:15:36.418190       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:15:36.418245       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:15:36.418276       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5f1d3684-1385-48b2-bf91-38e7276a92ef/globalmount with mount options([])
I1130 02:15:36.418285       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 111 lines ...
I1130 02:17:42.944115       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:17:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="2be6f372-7055-11ed-9767-000d3aad6ffc"
I1130 02:17:42.953157       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 8 milliseconds
I1130 02:17:42.953485       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="2be6f372-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9421383
I1130 02:17:57.473599       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:17:57.473644       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:17:57.473768       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:17:57.473856       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:17:57.473948       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:17:57.608136       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:18:02.571747       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:18:02.571867       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="2be59125-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7ec5deaa-5e58-4c47-9e57-cec49617d2d2" "latency"=19636774278
I1130 02:18:02.571915       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:18:02.571918       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2be59125-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19636924979
I1130 02:18:02.572003       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="2bd68957-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0d0b1869-2b12-4a94-bdd8-45ce7d20b30f" "latency"=19735430754
I1130 02:18:02.572061       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2bd68957-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19735574856
I1130 02:18:03.088495       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:18:03.088528       1 utils.go:79] GRPC request: {}
I1130 02:18:03.088583       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 112 lines ...
W1130 02:18:21.229172       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7ec5deaa-5e58-4c47-9e57-cec49617d2d2/globalmount" is not a mountpoint, deleting
I1130 02:18:21.229232       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7ec5deaa-5e58-4c47-9e57-cec49617d2d2/globalmount successfully
I1130 02:18:21.229246       1 utils.go:85] GRPC response: {}
I1130 02:18:27.474016       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:18:27.474042       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:18:27.474109       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:18:27.474185       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:18:27.608468       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:18:32.963916       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:18:32.964086       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="37e9c618-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f3cf2385-cb2f-4174-8d94-9a26f5bcd0d1" "latency"=29868649547
I1130 02:18:32.964207       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="37e9c618-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29868914549
I1130 02:18:33.733088       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:18:33.733122       1 utils.go:79] GRPC request: {}
I1130 02:18:33.733188       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:18:33.759147       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
... skipping 86 lines ...
I1130 02:19:53.583013       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:19:53.583032       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a6c8915d-bd61-48b0-81cc-9ab45e88d7fe/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a6c8915d-bd61-48b0-81cc-9ab45e88d7fe","csi.storage.k8s.io/pvc/name":"test.csi.azure.commngr9","csi.storage.k8s.io/pvc/namespace":"volume-5459","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-a6c8915d-bd61-48b0-81cc-9ab45e88d7fe"}
I1130 02:19:53.583211       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a6c8915d-bd61-48b0-81cc-9ab45e88d7fe-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:19:57.475899       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:57.476027       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:57.476099       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:57.476124       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:19:57.610488       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:20:02.627803       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:20:02.627902       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="79c4e6f4-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a6c8915d-bd61-48b0-81cc-9ab45e88d7fe" "latency"=9044597254
I1130 02:20:02.627950       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="79c4e6f4-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9044745556
I1130 02:20:03.463470       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 72 items received
I1130 02:20:03.466387       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=6832&timeout=8m33s&timeoutSeconds=513&watch=true 200 OK in 2 milliseconds
I1130 02:20:03.732894       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:20:03.732917       1 utils.go:79] GRPC request: {}
... skipping 117 lines ...
I1130 02:22:33.733323       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:22:42.944344       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:22:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="deb75a41-7055-11ed-9767-000d3aad6ffc"
I1130 02:22:42.949896       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 5 milliseconds
I1130 02:22:42.950138       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="deb75a41-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5847053
I1130 02:22:57.480622       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:57.480787       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:57.480806       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:22:57.480844       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:57.615382       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:03.732963       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:23:03.733010       1 utils.go:79] GRPC request: {}
I1130 02:23:03.733062       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:23:12.943579       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:23:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="f098e01a-7055-11ed-9767-000d3aad6ffc"
I1130 02:23:12.949318       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 5 milliseconds
I1130 02:23:12.950520       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="f098e01a-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6954963
I1130 02:23:27.481261       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:27.481305       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:27.481376       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:27.481445       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:23:27.615640       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:28.231338       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:23:28.231457       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="d5dee2f5-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e052e3a5-7412-417c-836b-66cb139514aa" "latency"=60127449578
I1130 02:23:28.231507       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d5dee2f5-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60127629080
I1130 02:23:29.755136       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:23:29.755158       1 utils.go:79] GRPC request: {}
I1130 02:23:29.755196       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:23:29.758837       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 61 lines ...
W1130 02:23:44.577179       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e052e3a5-7412-417c-836b-66cb139514aa/globalmount" is not a mountpoint, deleting
I1130 02:23:44.577272       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e052e3a5-7412-417c-836b-66cb139514aa/globalmount successfully
I1130 02:23:44.577302       1 utils.go:85] GRPC response: {}
I1130 02:23:57.481938       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:57.482007       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:57.481971       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:57.482179       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:23:57.616504       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:24:03.733243       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:24:03.733274       1 utils.go:79] GRPC request: {}
I1130 02:24:03.733345       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:24:12.944755       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:24:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="145c5310-7056-11ed-9767-000d3aad6ffc"
I1130 02:24:12.954855       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 9 milliseconds
I1130 02:24:12.955385       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="145c5310-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10680498
I1130 02:24:13.608179       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:24:13.608266       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="fa9f14ed-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-faf797e2-8f80-4865-a717-6792b58f87e6" "latency"=43846696154
I1130 02:24:13.608322       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="fa9f14ed-7055-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=43846878156
I1130 02:24:14.385915       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1130 02:24:14.386006       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:24:14.386068       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-faf797e2-8f80-4865-a717-6792b58f87e6/globalmount with mount options([])
I1130 02:24:14.386108       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 66 lines ...
I1130 02:24:42.944123       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:24:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="263dde53-7056-11ed-9767-000d3aad6ffc"
I1130 02:24:42.952888       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 8 milliseconds
I1130 02:24:42.953135       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="263dde53-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9053681
I1130 02:24:57.483940       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:24:57.484029       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:24:57.484064       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:24:57.484185       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:24:57.617548       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:25:03.732742       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:25:03.732785       1 utils.go:79] GRPC request: {}
I1130 02:25:03.732856       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:25:12.943520       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:25:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="381f6835-7056-11ed-9767-000d3aad6ffc"
I1130 02:25:12.952354       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 8 milliseconds
I1130 02:25:12.952848       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="381f6835-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9384386
I1130 02:25:27.484509       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:25:27.484540       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:25:27.484607       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:25:27.484713       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:25:27.617932       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:25:33.733151       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:25:33.733196       1 utils.go:79] GRPC request: {}
I1130 02:25:33.733259       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:25:34.752888       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:25:34.752979       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="1e7f63e5-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b3b8850c-75f1-49f2-95ae-49d543b64a7e" "latency"=64801162827
I1130 02:25:34.753014       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1e7f63e5-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=64801303728
I1130 02:25:36.771793       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1130 02:25:36.771835       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 02:25:36.771864       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b3b8850c-75f1-49f2-95ae-49d543b64a7e/globalmount with mount options([])
I1130 02:25:36.771881       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 113 lines ...
I1130 02:27:54.196969       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:27:54.196987       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-87d14b7f-5d22-4be2-85fb-a07a20ca91b2/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-87d14b7f-5d22-4be2-85fb-a07a20ca91b2","csi.storage.k8s.io/pvc/name":"test.csi.azure.com5drsx","csi.storage.k8s.io/pvc/namespace":"provisioning-9223","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-87d14b7f-5d22-4be2-85fb-a07a20ca91b2"}
I1130 02:27:54.197352       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-87d14b7f-5d22-4be2-85fb-a07a20ca91b2-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:27:57.488927       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:57.488998       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:57.489068       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:57.489104       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:27:57.622513       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:03.733639       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:28:03.733669       1 utils.go:79] GRPC request: {}
I1130 02:28:03.733742       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:28:12.944599       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:28:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="a3696543-7056-11ed-9767-000d3aad6ffc"
I1130 02:28:12.955174       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 10 milliseconds
I1130 02:28:12.955497       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="a3696543-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10951700
I1130 02:28:23.469612       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 196 items received
I1130 02:28:23.473616       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=11539&timeout=8m21s&timeoutSeconds=501&watch=true 200 OK in 3 milliseconds
I1130 02:28:27.489735       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:27.489763       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:27.489828       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:27.489994       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:28:27.623170       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:27.754599       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:28:27.754713       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="983ccc28-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-87d14b7f-5d22-4be2-85fb-a07a20ca91b2" "latency"=33557271885
I1130 02:28:27.754739       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="983ccc28-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=33557422485
I1130 02:28:29.391234       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 122 items received
I1130 02:28:29.393476       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=11672&timeout=8m36s&timeoutSeconds=516&watch=true 200 OK in 2 milliseconds
I1130 02:28:29.518489       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:28:29.518609       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
... skipping 78 lines ...
I1130 02:29:12.944718       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:29:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="c72caf84-7056-11ed-9767-000d3aad6ffc"
I1130 02:29:12.955772       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 10 milliseconds
I1130 02:29:12.957192       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="c72caf84-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12539115
I1130 02:29:27.492245       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:27.492276       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:27.492317       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:27.492383       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:29:27.624386       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:33.733813       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:29:33.733848       1 utils.go:79] GRPC request: {}
I1130 02:29:33.733889       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:29:37.895647       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:29:37.895811       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="c44da520-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c1fdf0e3-9ba3-4006-b860-51e6a3549076" "latency"=29768142422
I1130 02:29:37.895863       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c44da520-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29768374724
I1130 02:29:39.958370       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1130 02:29:39.958467       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:29:39.958520       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c1fdf0e3-9ba3-4006-b860-51e6a3549076/globalmount with mount options([])
I1130 02:29:39.958564       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 73 lines ...
I1130 02:30:37.725022       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:30:37.725041       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-7a847461-e81b-44e4-8351-a302f4c67e0e","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7a847461-e81b-44e4-8351-a302f4c67e0e","csi.storage.k8s.io/pvc/name":"test.csi.azure.comcvtmk","csi.storage.k8s.io/pvc/namespace":"volume-8663","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-7a847461-e81b-44e4-8351-a302f4c67e0e"}
I1130 02:30:37.725272       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7a847461-e81b-44e4-8351-a302f4c67e0e-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:30:42.943361       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:30:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="fcd16455-7056-11ed-9767-000d3aad6ffc"
I1130 02:30:42.957759       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 14 milliseconds
I1130 02:30:42.958130       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="fcd16455-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14814735
I1130 02:30:51.684898       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:30:51.684979       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="f9b52e3c-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7a847461-e81b-44e4-8351-a302f4c67e0e" "latency"=13959629156
I1130 02:30:51.685005       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f9b52e3c-7056-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13959752256
I1130 02:30:53.655092       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:30:53.655143       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:30:53.655156       1 utils.go:85] GRPC response: {}
I1130 02:30:53.660820       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 109 lines ...
I1130 02:33:33.733431       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:33:42.945022       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:33:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="681b7019-7057-11ed-9767-000d3aad6ffc"
I1130 02:33:42.953441       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 8 milliseconds
I1130 02:33:42.953657       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="681b7019-7057-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8846779
I1130 02:33:53.471559       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 176 items received
I1130 02:33:53.473989       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=14284&timeout=5m19s&timeoutSeconds=319&watch=true 200 OK in 2 milliseconds
I1130 02:33:53.671530       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:33:53.671599       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="60ece7e6-7057-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9bb452ff-6ed0-44fe-a848-6611767c8c1d" "latency"=22775713946
I1130 02:33:53.671625       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="60ece7e6-7057-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22775843148
I1130 02:33:53.676772       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:33:53.676852       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="60ecd872-7057-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9945cf1c-c264-4e67-a98c-e2e6c516b666" "latency"=22781353597
I1130 02:33:53.676896       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="60ecd872-7057-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22781502899
I1130 02:33:54.484093       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1130 02:33:54.484162       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:33:54.484181       1 utils.go:85] GRPC response: {}
I1130 02:33:54.484219       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
... skipping 119 lines ...
I1130 02:35:03.097433       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-21b5758a-f487-4eb9-9441-8d93f49924a9-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:35:03.097364       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-272e4590-4da1-4b22-9616-03b978206fba/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-272e4590-4da1-4b22-9616-03b978206fba","csi.storage.k8s.io/pvc/name":"test.csi.azure.comh2zdm","csi.storage.k8s.io/pvc/namespace":"multivolume-3081","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-272e4590-4da1-4b22-9616-03b978206fba"}
I1130 02:35:03.097575       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-272e4590-4da1-4b22-9616-03b978206fba-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:35:03.732864       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:35:03.732898       1 utils.go:79] GRPC request: {}
I1130 02:35:03.732951       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:35:11.096378       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:35:11.096513       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="97e1c7e4-7057-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-272e4590-4da1-4b22-9616-03b978206fba" "latency"=7998866146
I1130 02:35:11.096580       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="97e1c7e4-7057-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7998993247
I1130 02:35:11.096590       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:35:11.096751       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="97e1c176-7057-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-21b5758a-f487-4eb9-9441-8d93f49924a9" "latency"=7999220149
I1130 02:35:11.096802       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="97e1c176-7057-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7999386151
I1130 02:35:11.860719       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1130 02:35:11.860812       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:35:11.860902       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-21b5758a-f487-4eb9-9441-8d93f49924a9/globalmount with mount options([])
I1130 02:35:11.860936       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 336 lines ...
I1130 02:39:58.633955       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:39:58.633975       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-32de9938-4e0c-4b50-9d4d-a7be06d1535f/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-32de9938-4e0c-4b50-9d4d-a7be06d1535f","csi.storage.k8s.io/pvc/name":"pvc-jnrpl","csi.storage.k8s.io/pvc/namespace":"snapshotting-1502","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-32de9938-4e0c-4b50-9d4d-a7be06d1535f"}
I1130 02:39:58.634182       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-32de9938-4e0c-4b50-9d4d-a7be06d1535f-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:40:03.733346       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:40:03.733370       1 utils.go:79] GRPC request: {}
I1130 02:40:03.733418       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:40:09.106700       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:40:09.106800       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="48091666-7058-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-32de9938-4e0c-4b50-9d4d-a7be06d1535f" "latency"=10472525447
I1130 02:40:09.106857       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="48091666-7058-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10472687848
I1130 02:40:11.089948       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:40:11.090000       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:40:11.090065       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-32de9938-4e0c-4b50-9d4d-a7be06d1535f/globalmount with mount options([])
I1130 02:40:11.090092       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 206 lines ...
I1130 02:47:18.546679       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:47:18.546705       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e638880d-aabf-4244-87b0-4412db1c3847/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e638880d-aabf-4244-87b0-4412db1c3847","csi.storage.k8s.io/pvc/name":"pvc-spt6b","csi.storage.k8s.io/pvc/namespace":"snapshotting-5488","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-e638880d-aabf-4244-87b0-4412db1c3847"}
I1130 02:47:18.546961       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e638880d-aabf-4244-87b0-4412db1c3847-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:47:27.522491       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:27.522552       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:27.522614       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:27.522648       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:47:27.650179       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:33.732712       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:47:33.732736       1 utils.go:79] GRPC request: {}
I1130 02:47:33.732801       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:47:40.474387       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 85 items received
I1130 02:47:40.486571       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=21245&timeout=5m13s&timeoutSeconds=313&watch=true 200 OK in 12 milliseconds
I1130 02:47:42.943971       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:47:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="5cc920b7-7059-11ed-9767-000d3aad6ffc"
I1130 02:47:42.952638       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 8 milliseconds
I1130 02:47:42.952794       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="5cc920b7-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8907382
I1130 02:47:57.522993       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:57.523061       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:57.523118       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:57.523287       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:47:57.650445       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:03.732558       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:48:03.732588       1 utils.go:79] GRPC request: {}
I1130 02:48:03.732638       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:48:12.944121       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:48:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="6eaacb9f-7059-11ed-9767-000d3aad6ffc"
I1130 02:48:12.952546       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 8 milliseconds
I1130 02:48:12.952706       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="6eaacb9f-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8607978
I1130 02:48:27.524294       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:27.524352       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:27.524400       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:27.524501       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:48:27.650787       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:33.733290       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:48:33.733323       1 utils.go:79] GRPC request: {}
I1130 02:48:33.733470       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:48:42.944480       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:48:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="808c7b0e-7059-11ed-9767-000d3aad6ffc"
I1130 02:48:42.952510       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 7 milliseconds
I1130 02:48:42.952722       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="808c7b0e-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8317476
I1130 02:48:57.524708       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:57.524745       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:57.524755       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:57.524969       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:48:57.651020       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:03.732634       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:49:03.732665       1 utils.go:79] GRPC request: {}
I1130 02:49:03.732721       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:49:12.943905       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:49:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="926e0831-7059-11ed-9767-000d3aad6ffc"
I1130 02:49:12.952332       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 8 milliseconds
I1130 02:49:12.952922       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="926e0831-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9065682
E1130 02:49:18.540998       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="4e3e731e-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e638880d-aabf-4244-87b0-4412db1c3847" "latency"=119993829922
E1130 02:49:18.541072       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4e3e731e-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994124125
E1130 02:49:18.541097       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-e638880d-aabf-4244-87b0-4412db1c3847) to be attached to node (k8s-agentpool1-10232909-vmss000002): context canceled
I1130 02:49:19.051617       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:49:19.051643       1 utils.go:79] GRPC request: {}
I1130 02:49:19.051700       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:49:19.054975       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:49:19.054992       1 utils.go:79] GRPC request: {}
I1130 02:49:19.055021       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 3 lines ...
I1130 02:49:19.056888       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:49:19.056907       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e638880d-aabf-4244-87b0-4412db1c3847/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e638880d-aabf-4244-87b0-4412db1c3847","csi.storage.k8s.io/pvc/name":"pvc-spt6b","csi.storage.k8s.io/pvc/namespace":"snapshotting-5488","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-e638880d-aabf-4244-87b0-4412db1c3847"}
I1130 02:49:19.057103       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e638880d-aabf-4244-87b0-4412db1c3847-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:49:27.525211       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:27.525267       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:27.525328       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:27.525454       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:49:27.652151       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:33.733025       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:49:33.733089       1 utils.go:79] GRPC request: {}
I1130 02:49:33.733164       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:49:38.401947       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 59 items received
I1130 02:49:38.406287       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=21807&timeout=9m57s&timeoutSeconds=597&watch=true 200 OK in 4 milliseconds
I1130 02:49:42.944220       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:49:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="a44fb79e-7059-11ed-9767-000d3aad6ffc"
I1130 02:49:42.951600       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 7 milliseconds
I1130 02:49:42.951790       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="a44fb79e-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7627469
I1130 02:49:43.979432       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:49:43.979527       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="9612d6e5-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e638880d-aabf-4244-87b0-4412db1c3847" "latency"=24922332758
I1130 02:49:43.979587       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9612d6e5-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24922485260
I1130 02:49:44.753632       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:49:44.753656       1 utils.go:79] GRPC request: {}
I1130 02:49:44.753697       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:49:44.757659       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 36 lines ...
I1130 02:49:46.063387       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e638880d-aabf-4244-87b0-4412db1c3847/globalmount /var/lib/kubelet/pods/142b23d0-2a59-4581-a7be-b3e51236db06/volumes/kubernetes.io~csi/pvc-e638880d-aabf-4244-87b0-4412db1c3847/mount)
I1130 02:49:46.064907       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e638880d-aabf-4244-87b0-4412db1c3847/globalmount at /var/lib/kubelet/pods/142b23d0-2a59-4581-a7be-b3e51236db06/volumes/kubernetes.io~csi/pvc-e638880d-aabf-4244-87b0-4412db1c3847/mount successfully
I1130 02:49:46.064934       1 utils.go:85] GRPC response: {}
I1130 02:49:57.526275       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:57.526356       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:57.526486       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:57.526487       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:49:57.652791       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:03.733418       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:50:03.733467       1 utils.go:79] GRPC request: {}
I1130 02:50:03.733558       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:50:12.943707       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:50:12Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="b63146bc-7059-11ed-9767-000d3aad6ffc"
I1130 02:50:12.952083       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 8 milliseconds
... skipping 17 lines ...
W1130 02:50:21.538390       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e638880d-aabf-4244-87b0-4412db1c3847/globalmount" is not a mountpoint, deleting
I1130 02:50:21.538471       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e638880d-aabf-4244-87b0-4412db1c3847/globalmount successfully
I1130 02:50:21.538483       1 utils.go:85] GRPC response: {}
I1130 02:50:27.526892       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:27.526952       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:27.526999       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:27.527112       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:50:27.653517       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:33.733790       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:50:33.733814       1 utils.go:79] GRPC request: {}
I1130 02:50:33.733895       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:50:42.944282       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:50:42Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="c812ff92-7059-11ed-9767-000d3aad6ffc"
I1130 02:50:42.950316       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000002/status 200 OK in 5 milliseconds
I1130 02:50:42.952379       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="c812ff92-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8166074
I1130 02:50:44.477373       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:50:44.477460       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="a564c837-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-475def0d-f276-4f2e-a6d2-48e05f20e104" "latency"=59717412574
I1130 02:50:44.477485       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a564c837-7059-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59717557475
I1130 02:50:45.225148       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1130 02:50:45.225201       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:50:45.225236       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-475def0d-f276-4f2e-a6d2-48e05f20e104/globalmount with mount options([])
I1130 02:50:45.225250       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 163 lines ...
I1130 02:52:55.439009       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:52:55.439121       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b1d2ba7f-add7-436b-9f7b-83355be2da50/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b1d2ba7f-add7-436b-9f7b-83355be2da50","csi.storage.k8s.io/pvc/name":"test.csi.azure.com5ljsx","csi.storage.k8s.io/pvc/namespace":"multivolume-1708","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-b1d2ba7f-add7-436b-9f7b-83355be2da50"}
I1130 02:52:55.439479       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b1d2ba7f-add7-436b-9f7b-83355be2da50-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:52:57.530666       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:52:57.530697       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:52:57.530739       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:52:57.530794       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:52:57.657228       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:53:01.375004       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:53:01.375095       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="170c2981-705a-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b1d2ba7f-add7-436b-9f7b-83355be2da50" "latency"=5935483090
I1130 02:53:01.375126       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="170c2981-705a-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5935730792
I1130 02:53:03.248101       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:53:03.248153       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:53:03.248203       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b1d2ba7f-add7-436b-9f7b-83355be2da50/globalmount with mount options([])
I1130 02:53:03.248214       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 101 lines ...
I1130 02:55:18.226344       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:55:18.226367       1 utils.go:79] GRPC request: {}
I1130 02:55:18.226402       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:55:18.227894       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:55:18.227971       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9","csi.storage.k8s.io/pvc/name":"test.csi.azure.comlxdzz","csi.storage.k8s.io/pvc/namespace":"multivolume-1878","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9"}
I1130 02:55:18.228339       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 02:55:26.223968       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:55:26.224091       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="6c280263-705a-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dac85e23-b9ad-49a0-891c-14871ac0eae9" "latency"=7995619876
I1130 02:55:26.224144       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6c280263-705a-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7995873579
I1130 02:55:27.535032       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:27.535095       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:27.535136       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:27.661556       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 416 lines ...
I1130 03:11:59.389773       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:11:59.389793       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-04918b1e-398d-4c1c-a04c-d5123426551d/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-04918b1e-398d-4c1c-a04c-d5123426551d","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-nonroot-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-04918b1e-398d-4c1c-a04c-d5123426551d"}
I1130 03:11:59.390080       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-04918b1e-398d-4c1c-a04c-d5123426551d-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 03:12:03.733240       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:12:03.733267       1 utils.go:79] GRPC request: {}
I1130 03:12:03.733328       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:12:05.838338       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:12:05.838437       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="c0e52db2-705c-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-04918b1e-398d-4c1c-a04c-d5123426551d" "latency"=6448259830
I1130 03:12:05.838463       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c0e52db2-705c-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6448403431
I1130 03:12:07.771173       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 03:12:07.771254       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1130 03:12:07.771307       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-04918b1e-398d-4c1c-a04c-d5123426551d/globalmount with mount options([])
I1130 03:12:07.771317       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 47 lines ...
I1130 03:12:50.270361       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 03:12:50.270404       1 utils.go:79] GRPC request: {}
I1130 03:12:50.270430       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 03:12:50.271359       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:12:50.271380       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8b6147f6-2f70-43a1-b553-8f8c2f93d792/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8b6147f6-2f70-43a1-b553-8f8c2f93d792","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-g7v57-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-8b6147f6-2f70-43a1-b553-8f8c2f93d792"}
I1130 03:12:50.271745       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8b6147f6-2f70-43a1-b553-8f8c2f93d792-k8s-agentpool1-10232909-vmss000002-attachment)
I1130 03:12:56.576823       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:12:56.576910       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000002" "disk.csi.azure.com/request-id"="df391a54-705c-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8b6147f6-2f70-43a1-b553-8f8c2f93d792" "latency"=6305072450
I1130 03:12:56.576939       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="df391a54-705c-11ed-9767-000d3aad6ffc" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6305239751
I1130 03:12:57.361152       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1130 03:12:57.361236       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS
I1130 03:12:57.361267       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8b6147f6-2f70-43a1-b553-8f8c2f93d792/globalmount with mount options([])
I1130 03:12:57.361278       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 127 lines ...
I1130 02:12:23.650599       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=2567&timeout=8m40s&timeoutSeconds=520&watch=true 200 OK in 2 milliseconds
I1130 02:12:23.650611       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=2546&timeout=8m24s&timeoutSeconds=504&watch=true 200 OK in 2 milliseconds
I1130 02:12:23.744957       1 shared_informer.go:285] caches populated
I1130 02:12:23.744994       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb e2e-test
I1130 02:12:23.745006       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1130 02:12:23.747939       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I1130 02:12:23.748218       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1130 02:12:23.748229       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1130 02:12:23.748237       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1130 02:12:23.748262       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1130 02:12:23.749247       1 azure_auth.go:245] Using AzurePublicCloud environment
I1130 02:12:23.749261       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1130 02:12:23.749268       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 53 lines ...
I1130 02:12:23.881736       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes 201 Created in 4 milliseconds
I1130 02:12:23.887568       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 4 milliseconds
I1130 02:12:23.887765       1 azuredisk_v2.go:532]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="6db917ed-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).registerAzDriverNode" "latency"=11326162
I1130 02:12:23.887876       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:12:23Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="6dbad5a4-7054-11ed-bbee-000d3aad66aa"
I1130 02:12:23.888008       1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I1130 02:12:23.891357       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 409 Conflict in 3 milliseconds
E1130 02:12:23.891558       1 azuredisk_v2.go:591]  "msg"="Failed to update AzDriverNode status after creation" "error"="Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-10232909-vmss000000\": the object has been modified; please apply your changes to the latest version and try again" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="6dbad5a4-7054-11ed-bbee-000d3aad66aa"
E1130 02:12:23.891622       1 azuredisk_v2.go:543]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-10232909-vmss000000\": the object has been modified; please apply your changes to the latest version and try again]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="6dbad5a4-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=3768621
I1130 02:12:23.891631       1 azuredisk_v2.go:506] Starting heartbeat loop with initial delay 3.39s and frequency 30s
I1130 02:12:24.548354       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1130 02:12:24.548395       1 utils.go:79] GRPC request: {}
I1130 02:12:24.550272       1 utils.go:85] GRPC response: {"name":"test.csi.azure.com","vendor_version":"latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb"}
I1130 02:12:24.656140       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
I1130 02:12:24.656160       1 utils.go:79] GRPC request: {}
... skipping 13 lines ...
I1130 02:12:46.655856       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:12:46.655880       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705","csi.storage.k8s.io/pvc/name":"pvc-mds2q","csi.storage.k8s.io/pvc/namespace":"provisioning-2412","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705"}
I1130 02:12:46.656577       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:12:53.648996       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:12:53.649004       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:12:53.649020       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:12:53.649309       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:12:53.778318       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:12:57.283157       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:12:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="81a28ba9-7054-11ed-bbee-000d3aad66aa"
I1130 02:12:57.297687       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 14 milliseconds
I1130 02:12:57.298981       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="81a28ba9-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15887329
I1130 02:13:03.817934       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:13:03.817958       1 utils.go:79] GRPC request: {}
I1130 02:13:03.818077       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:13:23.650033       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:13:23.650094       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:13:23.650143       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:13:23.650168       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:13:23.778701       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:13:27.002171       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:13:27.002230       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="7b4d10a9-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705" "latency"=40345590502
I1130 02:13:27.002254       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7b4d10a9-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40345712609
I1130 02:13:27.282524       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:13:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="93841667-7054-11ed-bbee-000d3aad66aa"
I1130 02:13:27.289365       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 6 milliseconds
I1130 02:13:27.289602       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="93841667-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7131618
I1130 02:13:27.330879       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 62 lines ...
I1130 02:13:41.301091       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnstageVolume
I1130 02:13:41.301109       1 utils.go:79] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705"}
I1130 02:13:41.301160       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705
W1130 02:13:41.301545       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705" is not a mountpoint, deleting
I1130 02:13:41.301617       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-fbf2b426-c8fe-4ac9-a58b-6679664fa705 successfully
I1130 02:13:41.301638       1 utils.go:85] GRPC response: {}
I1130 02:13:42.343587       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:13:42.343670       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="93ba2203-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-48173db7-4a90-43bf-95a9-b401ab2b7de1" "latency"=14706927108
I1130 02:13:42.343711       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="93ba2203-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14707053115
I1130 02:13:42.350108       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:13:42.350174       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="94d5536c-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7d89dcc5-53e3-4a12-8a1f-ddd69341ed2f" "latency"=12857517514
I1130 02:13:42.350195       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="94d5536c-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12857617819
I1130 02:13:42.350266       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:13:42.350333       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="938c4fe4-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4586ac7d-80d6-4a12-9df9-808cdf96b473" "latency"=15013865664
I1130 02:13:42.350359       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="938c4fe4-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15014006072
I1130 02:13:43.005307       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdf under /dev/disk/azure/scsi1/
I1130 02:13:43.005310       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
I1130 02:13:43.005379       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 02:13:43.005318       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdd under /dev/disk/azure/scsi1/
... skipping 271 lines ...
I1130 02:14:33.817863       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:14:33.817894       1 utils.go:79] GRPC request: {}
I1130 02:14:33.817947       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:14:53.651714       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:14:53.651745       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:14:53.651734       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:14:53.651821       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:14:53.781153       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:14:57.284691       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:14:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="c9295425-7054-11ed-bbee-000d3aad66aa"
I1130 02:14:57.293589       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:14:57.293827       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="c9295425-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9179137
I1130 02:15:00.392498       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:15:00.392594       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="b76d743d-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-aa25bde2-1b04-4646-b5e0-f2129e06bf86" "latency"=32860389404
I1130 02:15:00.392640       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b76d743d-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=32860526712
I1130 02:15:01.030465       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:15:01.030521       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:15:01.030553       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-aa25bde2-1b04-4646-b5e0-f2129e06bf86/globalmount with mount options([])
I1130 02:15:01.030564       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 58 lines ...
I1130 02:15:20.380468       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:15:20.380488       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comr922c","csi.storage.k8s.io/pvc/namespace":"multivolume-6919","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6"}
I1130 02:15:20.380676       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:15:23.652510       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:15:23.652553       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:15:23.652498       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:15:23.652694       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:15:23.782101       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:15:26.253496       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:15:26.253532       1 utils.go:79] GRPC request: {}
I1130 02:15:26.253594       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:15:26.254574       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1130 02:15:26.254590       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-48173db7-4a90-43bf-95a9-b401ab2b7de1","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-48173db7-4a90-43bf-95a9-b401ab2b7de1/dev/9a9be33a-d683-472a-a683-6ca7869de7aa"}
I1130 02:15:26.255401       1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]}
I1130 02:15:27.282921       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:15:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="db0ab23c-7054-11ed-bbee-000d3aad66aa"
I1130 02:15:27.291762       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:15:27.291930       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="db0ab23c-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9055629
I1130 02:15:33.818952       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:15:33.818986       1 utils.go:79] GRPC request: {}
I1130 02:15:33.819041       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:15:51.748233       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:15:51.748293       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d6ed80cb-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6" "latency"=31367552486
I1130 02:15:51.748317       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d6ed80cb-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31367662792
I1130 02:15:52.384425       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1130 02:15:52.384483       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 02:15:52.384523       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6/globalmount with mount options([])
I1130 02:15:52.384533       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 43 lines ...
I1130 02:15:52.909315       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6/globalmount /var/lib/kubelet/pods/98e6f38d-1196-468b-8830-ba4227587ef3/volumes/kubernetes.io~csi/pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6/mount)
I1130 02:15:52.910546       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6/globalmount at /var/lib/kubelet/pods/98e6f38d-1196-468b-8830-ba4227587ef3/volumes/kubernetes.io~csi/pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6/mount successfully
I1130 02:15:52.910574       1 utils.go:85] GRPC response: {}
I1130 02:15:53.653465       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:15:53.653489       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:15:53.653514       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:15:53.653595       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:15:53.653614       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:15:53.782882       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:15:57.282231       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:15:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="ecec3958-7054-11ed-bbee-000d3aad66aa"
I1130 02:15:57.290838       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 02:15:57.292238       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="ecec3958-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10080889
I1130 02:15:57.470480       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:15:57.470503       1 utils.go:79] GRPC request: {}
... skipping 20 lines ...
W1130 02:16:01.730344       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6/globalmount" is not a mountpoint, deleting
I1130 02:16:01.730412       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d44cd21-2460-4316-880c-a3fbc7cb50f6/globalmount successfully
I1130 02:16:01.730425       1 utils.go:85] GRPC response: {}
I1130 02:16:03.817800       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:16:03.817832       1 utils.go:79] GRPC request: {}
I1130 02:16:03.817888       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:16:22.207241       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:16:22.207294       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="ea06e2b8-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a91c865f-2681-4bbb-adc0-60a9b4ed46dc" "latency"=29783505594
I1130 02:16:22.207317       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ea06e2b8-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29783605900
I1130 02:16:22.207534       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:16:22.207600       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="ea06c3e3-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9991dac3-0562-430a-9a6b-0f12196fba4e" "latency"=29784582657
I1130 02:16:22.207644       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ea06c3e3-7054-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29784710465
I1130 02:16:22.879756       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/
I1130 02:16:22.879809       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1130 02:16:22.879821       1 utils.go:85] GRPC response: {}
I1130 02:16:22.880447       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
... skipping 147 lines ...
I1130 02:16:53.191377       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0d0b1869-2b12-4a94-bdd8-45ce7d20b30f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-0d0b1869-2b12-4a94-bdd8-45ce7d20b30f","csi.storage.k8s.io/pvc/name":"test.csi.azure.comcxb5c","csi.storage.k8s.io/pvc/namespace":"multivolume-205","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-0d0b1869-2b12-4a94-bdd8-45ce7d20b30f"}
I1130 02:16:53.192333       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7ec5deaa-5e58-4c47-9e57-cec49617d2d2-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:16:53.192505       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-0d0b1869-2b12-4a94-bdd8-45ce7d20b30f-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:16:53.654498       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:16:53.654533       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:16:53.654555       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:16:53.654635       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:16:53.654700       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:16:53.783901       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:16:57.282240       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:16:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="10af7f12-7055-11ed-bbee-000d3aad66aa"
I1130 02:16:57.292609       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 10 milliseconds
I1130 02:16:57.292854       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="10af7f12-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10706426
I1130 02:16:57.470566       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:16:57.470599       1 utils.go:79] GRPC request: {}
... skipping 4 lines ...
I1130 02:16:57.473178       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:16:57.473199       1 utils.go:79] GRPC request: {}
I1130 02:16:57.473227       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:16:57.474146       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1130 02:16:57.474175       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-a91c865f-2681-4bbb-adc0-60a9b4ed46dc","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-a91c865f-2681-4bbb-adc0-60a9b4ed46dc/dev/36613bbd-e233-4443-a9f6-bf1b0130b498"}
I1130 02:16:57.474896       1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]}
I1130 02:16:57.550784       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:16:57.550882       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="0e3f7301-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0d0b1869-2b12-4a94-bdd8-45ce7d20b30f" "latency"=4358231317
I1130 02:16:57.550909       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0e3f7301-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4358525334
I1130 02:16:57.551085       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:16:57.551149       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="0e3f4ee3-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7ec5deaa-5e58-4c47-9e57-cec49617d2d2" "latency"=4358735246
I1130 02:16:57.551174       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0e3f4ee3-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4359715903
I1130 02:16:58.248547       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdc under /dev/disk/azure/scsi1/
I1130 02:16:58.248547       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdg under /dev/disk/azure/scsi1/
I1130 02:16:58.248616       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
I1130 02:16:58.248633       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
... skipping 171 lines ...
I1130 02:17:34.160433       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:17:34.160451       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ef31774d-2765-4dfd-b103-1c7d8c6416e2/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ef31774d-2765-4dfd-b103-1c7d8c6416e2","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzthfp","csi.storage.k8s.io/pvc/namespace":"multivolume-3195","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-ef31774d-2765-4dfd-b103-1c7d8c6416e2"}
I1130 02:17:34.160641       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ef31774d-2765-4dfd-b103-1c7d8c6416e2-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:17:53.655962       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:17:53.656013       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:17:53.655983       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:17:53.656094       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:17:53.785290       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:17:57.283126       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:17:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="3472e70b-7055-11ed-bbee-000d3aad66aa"
I1130 02:17:57.293778       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 10 milliseconds
I1130 02:17:57.293929       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="3472e70b-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10913436
I1130 02:18:03.817797       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:18:03.817887       1 utils.go:79] GRPC request: {}
I1130 02:18:03.818006       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:18:05.680252       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:18:05.680312       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="26aab4a1-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ef31774d-2765-4dfd-b103-1c7d8c6416e2" "latency"=31519608169
I1130 02:18:05.680336       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="26aab4a1-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31519719576
I1130 02:18:07.428202       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:18:07.428260       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:18:07.428294       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ef31774d-2765-4dfd-b103-1c7d8c6416e2/globalmount with mount options([])
I1130 02:18:07.428304       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 56 lines ...
W1130 02:18:16.368178       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ef31774d-2765-4dfd-b103-1c7d8c6416e2/globalmount" is not a mountpoint, deleting
I1130 02:18:16.368271       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ef31774d-2765-4dfd-b103-1c7d8c6416e2/globalmount successfully
I1130 02:18:16.368283       1 utils.go:85] GRPC response: {}
I1130 02:18:23.656685       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:18:23.656718       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:18:23.656872       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:18:23.656894       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:18:23.786199       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:18:27.283791       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:18:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="4654a5b0-7055-11ed-bbee-000d3aad66aa"
I1130 02:18:27.294414       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 9 milliseconds
I1130 02:18:27.294684       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="4654a5b0-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10939933
I1130 02:18:32.161013       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:18:32.161100       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="3edc1b8e-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-35578b73-33a5-4763-aa6e-1489596dacaf" "latency"=17411284281
I1130 02:18:32.161145       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3edc1b8e-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=17411438390
I1130 02:18:32.818378       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1130 02:18:32.818426       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:18:32.818457       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-35578b73-33a5-4763-aa6e-1489596dacaf/globalmount with mount options([nouuid])
I1130 02:18:32.818469       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 69 lines ...
I1130 02:19:11.761683       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:19:11.761702       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4f7ca2ac-d6c3-4a7e-b4fe-6574d8249930/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4f7ca2ac-d6c3-4a7e-b4fe-6574d8249930","csi.storage.k8s.io/pvc/name":"test.csi.azure.com9j4gw","csi.storage.k8s.io/pvc/namespace":"multivolume-8056","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-4f7ca2ac-d6c3-4a7e-b4fe-6574d8249930"}
I1130 02:19:11.761902       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4f7ca2ac-d6c3-4a7e-b4fe-6574d8249930-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:19:23.658516       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:23.658547       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:23.658617       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:23.658716       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:19:23.786965       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:27.284104       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:19:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="6a17f8dd-7055-11ed-bbee-000d3aad66aa"
I1130 02:19:27.291966       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 02:19:27.292214       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="6a17f8dd-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8147574
I1130 02:19:33.817789       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:19:33.817815       1 utils.go:79] GRPC request: {}
I1130 02:19:33.817861       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:19:53.659115       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:53.659215       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:53.659219       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:53.659304       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:19:53.788554       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:19:54.780191       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 25 items received
I1130 02:19:54.783415       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-10232909-vmss000000&resourceVersion=6715&timeout=6m48s&timeoutSeconds=408&watch=true 200 OK in 3 milliseconds
I1130 02:19:57.283247       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:19:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="7bf97a4f-7055-11ed-bbee-000d3aad66aa"
I1130 02:19:57.287945       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 4 milliseconds
I1130 02:19:57.289385       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="7bf97a4f-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6167960
I1130 02:20:03.643376       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:20:03.643459       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="60d77a15-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4f7ca2ac-d6c3-4a7e-b4fe-6574d8249930" "latency"=51881476150
I1130 02:20:03.643493       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="60d77a15-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=51881600958
I1130 02:20:03.817668       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:20:03.817696       1 utils.go:79] GRPC request: {}
I1130 02:20:03.817747       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:20:04.333363       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 77 lines ...
I1130 02:20:06.544268       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8c5c9447-23e9-45ff-bec0-b13594287d89-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:20:20.569232       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 126 items received
I1130 02:20:20.573214       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=7050&timeout=9m43s&timeoutSeconds=583&watch=true 200 OK in 3 milliseconds
I1130 02:20:23.659673       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:20:23.659752       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:20:23.659875       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:20:23.659983       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:20:23.660026       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:20:23.660094       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:20:23.660116       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:20:23.789427       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:20:27.283359       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:20:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8ddb213d-7055-11ed-bbee-000d3aad66aa"
I1130 02:20:27.290712       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 02:20:27.291145       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8ddb213d-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7818057
I1130 02:20:28.140843       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:20:28.140867       1 utils.go:79] GRPC request: {}
I1130 02:20:28.140915       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:20:28.141859       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1130 02:20:28.141876       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-4f7ca2ac-d6c3-4a7e-b4fe-6574d8249930","volume_path":"/var/lib/kubelet/pods/566b4566-de3d-467f-aef4-7a336741cb8f/volumes/kubernetes.io~csi/pvc-4f7ca2ac-d6c3-4a7e-b4fe-6574d8249930/mount"}
I1130 02:20:28.141984       1 utils.go:85] GRPC response: {"usage":[{"available":5179576320,"total":5196382208,"unit":1,"used":28672},{"available":327668,"total":327680,"unit":2,"used":12}]}
I1130 02:20:33.817631       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:20:33.817652       1 utils.go:79] GRPC request: {}
I1130 02:20:33.817703       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:20:34.962907       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:20:34.962980       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="803bddb1-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4459ec67-1052-4ee0-b4da-30f854b67d2a" "latency"=30533700842
I1130 02:20:34.963022       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="803bddb1-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30533858351
I1130 02:20:34.969163       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:20:34.969239       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="802e1397-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e556183e-3602-465e-bbdb-c7734f6cbc51" "latency"=30630350483
I1130 02:20:34.969279       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="802e1397-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30630479190
I1130 02:20:34.970675       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:20:34.970757       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="809819ce-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fa8a56e6-47f8-42c7-bada-49f4f3ff5aef" "latency"=29937021391
I1130 02:20:34.970791       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="809819ce-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29937151799
I1130 02:20:34.970871       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:20:34.970917       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="817e9a42-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8c5c9447-23e9-45ff-bec0-b13594287d89" "latency"=28426592458
I1130 02:20:34.970959       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="817e9a42-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28426705465
I1130 02:20:36.019691       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdg under /dev/disk/azure/scsi1/
I1130 02:20:36.019719       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sde under /dev/disk/azure/scsi1/
I1130 02:20:36.019751       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1130 02:20:36.019752       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
... skipping 363 lines ...
W1130 02:21:15.449009       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e556183e-3602-465e-bbdb-c7734f6cbc51/globalmount" is not a mountpoint, deleting
I1130 02:21:15.449071       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e556183e-3602-465e-bbdb-c7734f6cbc51/globalmount successfully
I1130 02:21:15.449081       1 utils.go:85] GRPC response: {}
I1130 02:21:23.661447       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:21:23.661537       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:21:23.661545       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:21:23.661636       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:21:23.789773       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:21:27.282511       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:21:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="b19e456f-7055-11ed-bbee-000d3aad66aa"
I1130 02:21:27.288494       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 5 milliseconds
I1130 02:21:27.288834       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="b19e456f-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6392176
I1130 02:21:28.518450       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:21:28.518516       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="a62ed272-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9614e0be-1135-485b-a96e-21379ebe8278" "latency"=20421339604
I1130 02:21:28.518540       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a62ed272-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20421441910
I1130 02:21:29.163109       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdh under /dev/disk/azure/scsi1/
I1130 02:21:29.163160       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType 
I1130 02:21:29.163193       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun5 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9614e0be-1135-485b-a96e-21379ebe8278/globalmount with mount options([])
I1130 02:21:29.163203       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun5" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun5])
... skipping 70 lines ...
I1130 02:22:19.463421       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:22:19.464108       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:22:19.464124       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e6d1175d-e447-4ed8-88d7-581be6f5938d/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e6d1175d-e447-4ed8-88d7-581be6f5938d","csi.storage.k8s.io/pvc/name":"test.csi.azure.com2zfz9","csi.storage.k8s.io/pvc/namespace":"volume-5337","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-e6d1175d-e447-4ed8-88d7-581be6f5938d"}
I1130 02:22:19.464279       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e6d1175d-e447-4ed8-88d7-581be6f5938d-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:22:23.663056       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:23.663093       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:23.663176       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:22:23.663205       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:23.791683       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:27.282370       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:22:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d56186ba-7055-11ed-bbee-000d3aad66aa"
I1130 02:22:27.291320       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:22:27.291567       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d56186ba-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9237441
I1130 02:22:33.818128       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:22:33.818158       1 utils.go:79] GRPC request: {}
I1130 02:22:33.818205       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:22:53.663336       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:53.663479       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:22:53.663493       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:53.663524       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:53.791927       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:22:57.283056       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:22:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e7434093-7055-11ed-bbee-000d3aad66aa"
I1130 02:22:57.295356       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 11 milliseconds
I1130 02:22:57.295860       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e7434093-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12952859
I1130 02:23:03.817502       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:23:03.817534       1 utils.go:79] GRPC request: {}
I1130 02:23:03.817600       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:23:23.663825       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:23.663933       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:23.663980       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:23.664059       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:23:23.792483       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:25.007372       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:23:25.007511       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d0b89637-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e6d1175d-e447-4ed8-88d7-581be6f5938d" "latency"=65543152777
I1130 02:23:25.007551       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d0b89637-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=65543284585
I1130 02:23:25.776044       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:23:25.776070       1 utils.go:79] GRPC request: {}
I1130 02:23:25.776114       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:23:25.780073       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 86 lines ...
W1130 02:23:35.857605       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e6d1175d-e447-4ed8-88d7-581be6f5938d/globalmount" is not a mountpoint, deleting
I1130 02:23:35.857675       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e6d1175d-e447-4ed8-88d7-581be6f5938d/globalmount successfully
I1130 02:23:35.857690       1 utils.go:85] GRPC response: {}
I1130 02:23:53.664495       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:53.664592       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:53.664654       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:53.664657       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:23:53.664711       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:23:53.664752       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:23:53.793099       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:23:57.282270       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:23:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="0b066c8e-7056-11ed-bbee-000d3aad66aa"
I1130 02:23:57.298266       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 15 milliseconds
I1130 02:23:57.298539       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="0b066c8e-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16253357
I1130 02:24:03.817653       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:24:03.817680       1 utils.go:79] GRPC request: {}
I1130 02:24:03.817721       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:24:15.478575       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:24:15.478632       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="f83fe524-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea3e3ea3-378e-427e-a9a0-1423ee819d1f" "latency"=49696383193
I1130 02:24:15.478656       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f83fe524-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=49696496801
I1130 02:24:15.486263       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:24:15.486333       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="f87d232d-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea2471cd-0602-4dd5-941f-8e17005c75e5" "latency"=49302731799
I1130 02:24:15.486377       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f87d232d-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=49302853506
I1130 02:24:15.486511       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:24:15.486576       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="f8e8b26f-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8be4b265-30b2-48f7-9d1a-fc8708d8be2f" "latency"=48598072079
I1130 02:24:15.486617       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f8e8b26f-7055-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=48598189986
I1130 02:24:16.145361       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdd under /dev/disk/azure/scsi1/
I1130 02:24:16.145361       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/
I1130 02:24:16.145419       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1130 02:24:16.145432       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
... skipping 176 lines ...
I1130 02:25:13.852918       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:25:13.852937       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8be4b265-30b2-48f7-9d1a-fc8708d8be2f/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8be4b265-30b2-48f7-9d1a-fc8708d8be2f","csi.storage.k8s.io/pvc/name":"test.csi.azure.comx5z4t","csi.storage.k8s.io/pvc/namespace":"snapshotting-1971","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-8be4b265-30b2-48f7-9d1a-fc8708d8be2f"}
I1130 02:25:13.853133       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8be4b265-30b2-48f7-9d1a-fc8708d8be2f-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:25:23.666317       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:25:23.666344       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:25:23.666375       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:25:23.666461       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:25:23.794877       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:25:27.284806       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:25:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="40abb7db-7056-11ed-bbee-000d3aad66aa"
I1130 02:25:27.298070       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 12 milliseconds
I1130 02:25:27.298702       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="40abb7db-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13941517
I1130 02:25:33.470163       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:25:33.470248       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="38aa368a-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8be4b265-30b2-48f7-9d1a-fc8708d8be2f" "latency"=19617046981
I1130 02:25:33.470285       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="38aa368a-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19617167788
I1130 02:25:33.817846       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:25:33.817871       1 utils.go:79] GRPC request: {}
I1130 02:25:33.817925       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:25:35.237708       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
... skipping 61 lines ...
I1130 02:25:44.165230       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:25:44.165231       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:25:44.165260       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-c1a75266-8c05-4934-86f8-9b2f94efb338","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c1a75266-8c05-4934-86f8-9b2f94efb338","csi.storage.k8s.io/pvc/name":"inline-volume-tester-v7sdf-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-947","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-c1a75266-8c05-4934-86f8-9b2f94efb338"}
I1130 02:25:44.165360       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-824bab4e-5712-4a81-8f9a-df49479bf159","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-824bab4e-5712-4a81-8f9a-df49479bf159","csi.storage.k8s.io/pvc/name":"inline-volume-tester-v7sdf-my-volume-1","csi.storage.k8s.io/pvc/namespace":"ephemeral-947","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-824bab4e-5712-4a81-8f9a-df49479bf159"}
I1130 02:25:44.165532       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c1a75266-8c05-4934-86f8-9b2f94efb338-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:25:44.165552       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-824bab4e-5712-4a81-8f9a-df49479bf159-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:25:50.858196       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:25:50.858262       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="4abb8550-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-824bab4e-5712-4a81-8f9a-df49479bf159" "latency"=6692635953
I1130 02:25:50.858286       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4abb8550-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6692760660
I1130 02:25:50.867770       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:25:50.867839       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="4abb842e-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c1a75266-8c05-4934-86f8-9b2f94efb338" "latency"=6702221416
I1130 02:25:50.867876       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4abb842e-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=6702373824
I1130 02:25:51.548872       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1130 02:25:51.548926       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 02:25:51.548941       1 utils.go:85] GRPC response: {}
I1130 02:25:51.549322       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
... skipping 56 lines ...
I1130 02:25:57.478342       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1130 02:25:57.478361       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-c1a75266-8c05-4934-86f8-9b2f94efb338","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-c1a75266-8c05-4934-86f8-9b2f94efb338/dev/6cfa9622-c02b-4f26-9db3-c4d70382363e"}
I1130 02:25:57.479085       1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]}
I1130 02:26:03.819130       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:26:03.819163       1 utils.go:79] GRPC request: {}
I1130 02:26:03.819221       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:26:16.499871       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:26:16.499978       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="50c90cc2-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-54f3a0f7-adc6-4aa5-b8c1-47c1bc805fe3" "latency"=22179331454
I1130 02:26:16.500018       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="50c90cc2-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22179494763
I1130 02:26:17.129084       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
I1130 02:26:17.129153       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1130 02:26:17.129198       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-54f3a0f7-adc6-4aa5-b8c1-47c1bc805fe3/globalmount with mount options([])
I1130 02:26:17.129224       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 34 lines ...
I1130 02:26:17.789011       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f06c5d81-9038-40bb-845b-37ac737142be","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f06c5d81-9038-40bb-845b-37ac737142be","csi.storage.k8s.io/pvc/name":"inline-volume-tester-g9zf2-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-4900","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-f06c5d81-9038-40bb-845b-37ac737142be"}
I1130 02:26:17.789226       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-f06c5d81-9038-40bb-845b-37ac737142be-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:26:23.654010       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 148 items received
I1130 02:26:23.657296       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=10589&timeout=9m52s&timeoutSeconds=592&watch=true 200 OK in 3 milliseconds
I1130 02:26:23.668546       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:26:23.668646       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:26:23.668688       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:26:23.668693       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:26:23.797057       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:26:26.922086       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1130 02:26:26.922109       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-824bab4e-5712-4a81-8f9a-df49479bf159/6cfa9622-c02b-4f26-9db3-c4d70382363e","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-824bab4e-5712-4a81-8f9a-df49479bf159"}
I1130 02:26:26.922180       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-824bab4e-5712-4a81-8f9a-df49479bf159 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-824bab4e-5712-4a81-8f9a-df49479bf159/6cfa9622-c02b-4f26-9db3-c4d70382363e
I1130 02:26:26.922217       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-824bab4e-5712-4a81-8f9a-df49479bf159/6cfa9622-c02b-4f26-9db3-c4d70382363e" is a mountpoint, unmounting
... skipping 30 lines ...
I1130 02:26:27.283025       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:26:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="646eb87c-7056-11ed-bbee-000d3aad66aa"
I1130 02:26:27.297091       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 13 milliseconds
I1130 02:26:27.299001       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="646eb87c-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=16018627
I1130 02:26:33.817996       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:26:33.818024       1 utils.go:79] GRPC request: {}
I1130 02:26:33.818080       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:26:36.868836       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:26:36.868912       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="5ec6162c-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f06c5d81-9038-40bb-845b-37ac737142be" "latency"=19079615988
I1130 02:26:36.868944       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5ec6162c-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19079736894
I1130 02:26:37.562176       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdf under /dev/disk/azure/scsi1/
I1130 02:26:37.562234       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:26:37.562247       1 utils.go:85] GRPC response: {}
I1130 02:26:37.569911       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 75 lines ...
W1130 02:27:13.334736       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f06c5d81-9038-40bb-845b-37ac737142be" is not a mountpoint, deleting
I1130 02:27:13.334813       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f06c5d81-9038-40bb-845b-37ac737142be successfully
I1130 02:27:13.334826       1 utils.go:85] GRPC response: {}
I1130 02:27:23.669579       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:23.669668       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:23.669720       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:23.669783       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:27:23.798013       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:27.283088       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:27:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="88320012-7056-11ed-bbee-000d3aad66aa"
I1130 02:27:27.293344       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 10 milliseconds
I1130 02:27:27.293543       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="88320012-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10520015
I1130 02:27:32.822641       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:27:32.822727       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="7561adee-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c1c0c695-b863-4f38-80d7-f52c018a6961" "latency"=37103846477
I1130 02:27:32.822763       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7561adee-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=37103983885
I1130 02:27:33.501750       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:27:33.501778       1 utils.go:79] GRPC request: {}
I1130 02:27:33.501834       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:27:33.504651       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdd under /dev/disk/azure/scsi1/
... skipping 40 lines ...
I1130 02:27:34.006602       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c1c0c695-b863-4f38-80d7-f52c018a6961/globalmount /var/lib/kubelet/pods/1e940b0f-cc07-4874-aa82-365b1aaeba01/volumes/kubernetes.io~csi/pvc-c1c0c695-b863-4f38-80d7-f52c018a6961/mount)
I1130 02:27:34.007531       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c1c0c695-b863-4f38-80d7-f52c018a6961/globalmount at /var/lib/kubelet/pods/1e940b0f-cc07-4874-aa82-365b1aaeba01/volumes/kubernetes.io~csi/pvc-c1c0c695-b863-4f38-80d7-f52c018a6961/mount successfully
I1130 02:27:34.007552       1 utils.go:85] GRPC response: {}
I1130 02:27:53.670252       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:53.670300       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:53.670317       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:53.670402       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:27:53.798693       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:27:57.282297       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:27:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="9a13857e-7056-11ed-bbee-000d3aad66aa"
I1130 02:27:57.288580       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 6 milliseconds
I1130 02:27:57.288823       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="9a13857e-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6557485
I1130 02:28:03.818260       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:28:03.818291       1 utils.go:79] GRPC request: {}
... skipping 19 lines ...
W1130 02:28:10.662944       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c1c0c695-b863-4f38-80d7-f52c018a6961/globalmount" is not a mountpoint, deleting
I1130 02:28:10.663025       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c1c0c695-b863-4f38-80d7-f52c018a6961/globalmount successfully
I1130 02:28:10.663044       1 utils.go:85] GRPC response: {}
I1130 02:28:23.671336       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:23.671440       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:23.671446       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:23.671569       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:28:23.799833       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:27.283157       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:28:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="abf54987-7056-11ed-bbee-000d3aad66aa"
I1130 02:28:27.291823       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:28:27.292116       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="abf54987-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9005818
I1130 02:28:33.818286       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:28:33.818319       1 utils.go:79] GRPC request: {}
I1130 02:28:33.818377       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:28:53.671674       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:53.671740       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:53.671781       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:53.671809       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:28:53.800056       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:28:57.285683       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:28:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="bdd74f59-7056-11ed-bbee-000d3aad66aa"
I1130 02:28:57.295451       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 9 milliseconds
I1130 02:28:57.295691       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="bdd74f59-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10035687
I1130 02:29:03.817621       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:29:03.817651       1 utils.go:79] GRPC request: {}
I1130 02:29:03.817708       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:29:23.671843       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:23.671967       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:23.671989       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:23.672039       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:29:23.800357       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:27.282428       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:29:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="cfb87197-7056-11ed-bbee-000d3aad66aa"
I1130 02:29:27.290519       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 02:29:27.291855       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="cfb87197-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9500056
E1130 02:29:33.501092       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8be7d8ab-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722" "latency"=119993005453
E1130 02:29:33.501194       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8be7d8ab-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119993224265
E1130 02:29:33.501214       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722) to be attached to node (k8s-agentpool1-10232909-vmss000000): context canceled
I1130 02:29:33.818519       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:29:33.818547       1 utils.go:79] GRPC request: {}
I1130 02:29:33.818606       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:29:34.050318       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:29:34.050341       1 utils.go:79] GRPC request: {}
I1130 02:29:34.050380       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
... skipping 3 lines ...
I1130 02:29:34.055055       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:29:34.055071       1 utils.go:79] GRPC request: {}
I1130 02:29:34.055203       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:29:34.056033       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:29:34.056049       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722","csi.storage.k8s.io/pvc/name":"test.csi.azure.comrb4mp","csi.storage.k8s.io/pvc/namespace":"multivolume-7029","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722"}
I1130 02:29:34.056253       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:29:36.844316       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:29:36.844380       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d3c20f39-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722" "latency"=2788059707
I1130 02:29:36.844406       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d3c20f39-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2788168313
I1130 02:29:38.723838       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1130 02:29:38.723904       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:29:38.723938       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722/globalmount with mount options([nouuid])
I1130 02:29:38.723956       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 54 lines ...
I1130 02:29:47.347861       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722/globalmount" is a mountpoint, unmounting
I1130 02:29:47.347876       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722/globalmount
W1130 02:29:47.360999       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722/globalmount" is not a mountpoint, deleting
I1130 02:29:47.361073       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9e9bab4f-6dc4-4b41-9e0b-da869908e722/globalmount successfully
I1130 02:29:47.361084       1 utils.go:85] GRPC response: {}
I1130 02:29:53.672022       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:53.672147       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:29:53.672173       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:53.672187       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:53.801539       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:29:57.282729       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:29:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e19a21bd-7056-11ed-bbee-000d3aad66aa"
I1130 02:29:57.291296       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:29:57.291630       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e19a21bd-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8968125
I1130 02:30:02.189586       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:30:02.189676       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d7c7c6dd-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-87093b91-c968-4b17-984e-5c9560aacac2" "latency"=21384976886
I1130 02:30:02.189706       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d7c7c6dd-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21385108394
I1130 02:30:02.901111       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1130 02:30:02.901182       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:30:02.901598       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-87093b91-c968-4b17-984e-5c9560aacac2/globalmount with mount options([nouuid])
I1130 02:30:02.901627       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 103 lines ...
I1130 02:30:21.135286       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-87093b91-c968-4b17-984e-5c9560aacac2/globalmount /var/lib/kubelet/pods/9550ce61-b8a2-4388-bd6b-a18aa01f768e/volumes/kubernetes.io~csi/pvc-87093b91-c968-4b17-984e-5c9560aacac2/mount)
I1130 02:30:21.136240       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-87093b91-c968-4b17-984e-5c9560aacac2/globalmount at /var/lib/kubelet/pods/9550ce61-b8a2-4388-bd6b-a18aa01f768e/volumes/kubernetes.io~csi/pvc-87093b91-c968-4b17-984e-5c9560aacac2/mount successfully
I1130 02:30:21.136260       1 utils.go:85] GRPC response: {}
I1130 02:30:23.672716       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:30:23.672750       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:30:23.672729       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:30:23.672889       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:30:23.802070       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:30:27.282334       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:30:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="f37bb554-7056-11ed-bbee-000d3aad66aa"
I1130 02:30:27.290601       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:30:27.290883       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="f37bb554-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8586789
I1130 02:30:33.817832       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:30:33.817853       1 utils.go:79] GRPC request: {}
... skipping 14 lines ...
I1130 02:30:36.503685       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-87093b91-c968-4b17-984e-5c9560aacac2/globalmount
I1130 02:30:36.503723       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-87093b91-c968-4b17-984e-5c9560aacac2/globalmount" is a mountpoint, unmounting
I1130 02:30:36.503739       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-87093b91-c968-4b17-984e-5c9560aacac2/globalmount
W1130 02:30:36.511528       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-87093b91-c968-4b17-984e-5c9560aacac2/globalmount" is not a mountpoint, deleting
I1130 02:30:36.511612       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-87093b91-c968-4b17-984e-5c9560aacac2/globalmount successfully
I1130 02:30:36.511640       1 utils.go:85] GRPC response: {}
I1130 02:30:47.954375       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:30:47.954507       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e52c33e5-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3c84aa97-2d71-4966-b5ae-db563477c5fe" "latency"=44681218901
I1130 02:30:47.954556       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e52c33e5-7056-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=44681422713
I1130 02:30:48.615376       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1130 02:30:48.615424       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 02:30:48.615455       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c84aa97-2d71-4966-b5ae-db563477c5fe/globalmount with mount options([])
I1130 02:30:48.615466       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 97 lines ...
I1130 02:32:41.816115       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:32:41.816950       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:32:41.816968       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a78d770d-a220-43bd-b8e0-743408c5ce37/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a78d770d-a220-43bd-b8e0-743408c5ce37","csi.storage.k8s.io/pvc/name":"test.csi.azure.com88q9j","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-6774","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-a78d770d-a220-43bd-b8e0-743408c5ce37"}
I1130 02:32:41.817576       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a78d770d-a220-43bd-b8e0-743408c5ce37-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:32:53.676824       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:32:53.676875       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:32:53.676941       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:32:53.676943       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:32:53.806331       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:32:57.282428       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:32:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="4ce3e887-7057-11ed-bbee-000d3aad66aa"
I1130 02:32:57.294204       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 11 milliseconds
I1130 02:32:57.294480       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="4ce3e887-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12071506
I1130 02:33:03.818120       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:33:03.818148       1 utils.go:79] GRPC request: {}
I1130 02:33:03.818204       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:33:21.860904       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:33:21.860983       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="43ac27a8-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a78d770d-a220-43bd-b8e0-743408c5ce37" "latency"=40043307241
I1130 02:33:21.861016       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="43ac27a8-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40043509853
I1130 02:33:22.207737       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:33:22.207759       1 utils.go:79] GRPC request: {}
I1130 02:33:22.207797       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:33:22.211109       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 11 lines ...
I1130 02:33:23.630390       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
I1130 02:33:23.656602       1 mount_linux.go:490] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n"
I1130 02:33:23.656629       1 mount_linux.go:376] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun0
I1130 02:33:23.677693       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:33:23.677823       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:33:23.677833       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:33:23.677919       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:33:23.688480       1 mount_linux.go:477] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a78d770d-a220-43bd-b8e0-743408c5ce37/globalmount
I1130 02:33:23.688516       1 mount_linux.go:183] Mounting cmd (mount) with arguments (-t ext4 -o defaults /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a78d770d-a220-43bd-b8e0-743408c5ce37/globalmount)
I1130 02:33:23.692179       1 nodeserver_v2.go:215] NodeStageVolume: format /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a78d770d-a220-43bd-b8e0-743408c5ce37/globalmount successfully.
I1130 02:33:23.692891       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
I1130 02:33:23.697922       1 mount_linux.go:490] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n"
I1130 02:33:23.697953       1 resizefs_linux.go:124] ResizeFs.needResize - checking mounted volume /dev/disk/azure/scsi1/lun0
... skipping 39 lines ...
W1130 02:33:30.780213       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a78d770d-a220-43bd-b8e0-743408c5ce37/globalmount" is not a mountpoint, deleting
I1130 02:33:30.780306       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a78d770d-a220-43bd-b8e0-743408c5ce37/globalmount successfully
I1130 02:33:30.780326       1 utils.go:85] GRPC response: {}
I1130 02:33:33.818230       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:33:33.818263       1 utils.go:79] GRPC request: {}
I1130 02:33:33.818318       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:33:52.334564       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:33:52.334642       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="5bc00577-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6845e7bf-984a-4b4b-b9c7-f9bec72bc7ff" "latency"=30121512692
I1130 02:33:52.334668       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5bc00577-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30121652200
I1130 02:33:53.026928       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1130 02:33:53.026984       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:33:53.027018       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6845e7bf-984a-4b4b-b9c7-f9bec72bc7ff/globalmount with mount options([])
I1130 02:33:53.027029       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 36 lines ...
I1130 02:33:53.548660       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:33:53.549380       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:33:53.549399       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a","csi.storage.k8s.io/pvc/name":"test.csi.azure.com852l4","csi.storage.k8s.io/pvc/namespace":"provisioning-2355","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a"}
I1130 02:33:53.549576       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:33:53.678444       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:33:53.678499       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:33:53.678561       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:33:53.678609       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:33:53.806910       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:33:57.282287       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:33:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="70a727c7-7057-11ed-bbee-000d3aad66aa"
I1130 02:33:57.290967       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:33:57.291557       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="70a727c7-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9332149
I1130 02:33:57.673004       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
... skipping 18 lines ...
I1130 02:34:03.817896       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:34:03.817927       1 utils.go:79] GRPC request: {}
I1130 02:34:03.818006       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:34:23.678714       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:34:23.678759       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:34:23.678789       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:34:23.678888       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:34:23.807205       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:34:27.284156       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:34:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="828914aa-7057-11ed-bbee-000d3aad66aa"
I1130 02:34:27.296394       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 11 milliseconds
I1130 02:34:27.297001       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="828914aa-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12888154
I1130 02:34:33.818423       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:34:33.818456       1 utils.go:79] GRPC request: {}
I1130 02:34:33.818514       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:34:48.100808       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:34:48.100866       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="6e6d991f-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a" "latency"=54551225667
I1130 02:34:48.100888       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6e6d991f-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=54551331773
I1130 02:34:48.771840       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1130 02:34:48.771878       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 02:34:48.771911       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a/globalmount with mount options([])
I1130 02:34:48.771923       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 45 lines ...
I1130 02:34:49.358944       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9bb452ff-6ed0-44fe-a848-6611767c8c1d-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:34:51.659759       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 63 items received
I1130 02:34:51.663032       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=14705&timeout=6m30s&timeoutSeconds=390&watch=true 200 OK in 3 milliseconds
I1130 02:34:53.679524       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:34:53.679578       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:34:53.679610       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:34:53.679658       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:34:53.679704       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:34:53.808033       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:34:54.406958       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1130 02:34:54.406983       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/f828252b-71ba-40f0-84e6-9d782e67661d/volumes/kubernetes.io~csi/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a"}
I1130 02:34:54.407044       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a on /var/lib/kubelet/pods/f828252b-71ba-40f0-84e6-9d782e67661d/volumes/kubernetes.io~csi/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a/mount
I1130 02:34:54.407076       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/f828252b-71ba-40f0-84e6-9d782e67661d/volumes/kubernetes.io~csi/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a/mount" is a mountpoint, unmounting
I1130 02:34:54.407086       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/f828252b-71ba-40f0-84e6-9d782e67661d/volumes/kubernetes.io~csi/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a/mount
... skipping 17 lines ...
I1130 02:35:03.817412       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:35:03.817437       1 utils.go:79] GRPC request: {}
I1130 02:35:03.817494       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:35:23.680019       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:35:23.680102       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:35:23.680140       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:35:23.680215       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:35:23.680239       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:35:23.808610       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:35:27.283730       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:35:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="a64c4a70-7057-11ed-bbee-000d3aad66aa"
I1130 02:35:27.291166       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 02:35:27.291716       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="a64c4a70-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8011169
I1130 02:35:32.786348       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 27 items received
I1130 02:35:32.791438       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-10232909-vmss000000&resourceVersion=15161&timeout=8m6s&timeoutSeconds=486&watch=true 200 OK in 4 milliseconds
I1130 02:35:33.817785       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:35:33.817811       1 utils.go:79] GRPC request: {}
I1130 02:35:33.817856       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:35:49.676294       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:35:49.676357       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8fb12316-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9945cf1c-c264-4e67-a98c-e2e6c516b666" "latency"=60319247558
I1130 02:35:49.676390       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8fb12316-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60319391567
I1130 02:35:49.680542       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:35:49.680615       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8fb16e94-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9bb452ff-6ed0-44fe-a848-6611767c8c1d" "latency"=60321605296
I1130 02:35:49.680653       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8fb16e94-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60321717703
I1130 02:35:50.349064       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:35:50.349115       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:35:50.349127       1 utils.go:85] GRPC response: {}
I1130 02:35:50.352804       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 36 lines ...
I1130 02:35:51.213285       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:35:51.214187       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:35:51.214204       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a","csi.storage.k8s.io/pvc/name":"test.csi.azure.com852l4","csi.storage.k8s.io/pvc/namespace":"provisioning-2355","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a"}
I1130 02:35:51.214355       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:35:53.680730       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:35:53.680786       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:35:53.680870       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:35:53.680766       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:35:53.809572       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:35:57.283214       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:35:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="b82dd7e3-7057-11ed-bbee-000d3aad66aa"
I1130 02:35:57.291490       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:35:57.291772       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="b82dd7e3-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8628206
I1130 02:36:03.817527       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 35 lines ...
I1130 02:36:07.016861       1 utils.go:85] GRPC response: {}
I1130 02:36:15.658617       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 175 items received
I1130 02:36:15.661828       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=15568&timeout=7m44s&timeoutSeconds=464&watch=true 200 OK in 3 milliseconds
I1130 02:36:23.681433       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:36:23.681524       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:36:23.681583       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:36:23.681695       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:36:23.809851       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:36:27.283121       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:36:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="ca0f7867-7057-11ed-bbee-000d3aad66aa"
I1130 02:36:27.294015       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 10 milliseconds
I1130 02:36:27.294262       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="ca0f7867-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11171654
I1130 02:36:30.169491       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:36:30.169581       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="b48fd188-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a" "latency"=38955160491
I1130 02:36:30.169624       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b48fd188-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=38955274398
I1130 02:36:30.836345       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
I1130 02:36:30.836399       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:36:30.836428       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3a7e74e9-1c27-4242-ae0f-80d31b3eff8a/globalmount with mount options([])
I1130 02:36:30.836438       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1])
... skipping 75 lines ...
I1130 02:37:39.246035       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:37:39.246055       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-bb7b1147-2c0d-4502-ae22-f57bc5bf4e24","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-bb7b1147-2c0d-4502-ae22-f57bc5bf4e24","csi.storage.k8s.io/pvc/name":"test.csi.azure.comvqgqh","csi.storage.k8s.io/pvc/namespace":"multivolume-2401","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-bb7b1147-2c0d-4502-ae22-f57bc5bf4e24"}
I1130 02:37:39.246281       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-bb7b1147-2c0d-4502-ae22-f57bc5bf4e24-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:37:53.683168       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:37:53.683228       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:37:53.683419       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:37:53.683510       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:37:53.811732       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:37:57.282699       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:37:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="ffb4507e-7057-11ed-bbee-000d3aad66aa"
I1130 02:37:57.289793       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 6 milliseconds
I1130 02:37:57.292607       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="ffb4507e-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9959283
I1130 02:38:03.817543       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:38:03.817570       1 utils.go:79] GRPC request: {}
I1130 02:38:03.817626       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:38:09.068985       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:38:09.069056       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="f4f42e69-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-bb7b1147-2c0d-4502-ae22-f57bc5bf4e24" "latency"=29822687449
I1130 02:38:09.069082       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f4f42e69-7057-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29822823257
I1130 02:38:09.342816       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:38:09.342839       1 utils.go:79] GRPC request: {}
I1130 02:38:09.342880       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:38:09.346849       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 52 lines ...
I1130 02:38:21.225760       1 utils.go:85] GRPC response: {}
I1130 02:38:21.665625       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 209 items received
I1130 02:38:21.668285       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=16796&timeout=6m40s&timeoutSeconds=400&watch=true 200 OK in 2 milliseconds
I1130 02:38:23.683817       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:38:23.683821       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:38:23.683830       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:38:23.684074       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:38:23.684213       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:38:23.812201       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:38:27.282236       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:38:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="1195e1f5-7058-11ed-bbee-000d3aad66aa"
I1130 02:38:27.288270       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 5 milliseconds
I1130 02:38:27.288844       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="1195e1f5-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6644289
I1130 02:38:29.757661       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:38:29.757738       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="06e57d1b-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-570ffe9f-5d9f-491e-8139-c433673a1d74" "latency"=20408677525
I1130 02:38:29.757775       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="06e57d1b-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20408808933
I1130 02:38:29.759493       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:38:29.759544       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="06f2e65b-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5a65833b-cf4c-43c8-9c46-6f7ce10b70e1" "latency"=20322634593
I1130 02:38:29.759571       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="06f2e65b-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20322725698
I1130 02:38:30.441435       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/
I1130 02:38:30.441503       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:38:30.441550       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-570ffe9f-5d9f-491e-8139-c433673a1d74/globalmount with mount options([nouuid])
I1130 02:38:30.441577       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 107 lines ...
I1130 02:38:47.323012       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5a65833b-cf4c-43c8-9c46-6f7ce10b70e1/globalmount successfully
I1130 02:38:47.323027       1 utils.go:85] GRPC response: {}
I1130 02:38:47.323072       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-570ffe9f-5d9f-491e-8139-c433673a1d74/globalmount successfully
I1130 02:38:47.323093       1 utils.go:85] GRPC response: {}
I1130 02:38:53.684311       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:38:53.684421       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:38:53.684445       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:38:53.684469       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:38:53.812831       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:38:57.282370       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:38:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="23778944-7058-11ed-bbee-000d3aad66aa"
I1130 02:38:57.294000       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 11 milliseconds
I1130 02:38:57.294242       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="23778944-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11932295
I1130 02:39:03.819747       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:39:03.819776       1 utils.go:79] GRPC request: {}
I1130 02:39:03.819829       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:39:23.684935       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:39:23.685032       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:39:23.685081       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:39:23.685147       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:39:23.813386       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:39:27.282150       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:39:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="355924df-7058-11ed-bbee-000d3aad66aa"
I1130 02:39:27.291260       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:39:27.291562       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="355924df-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9433650
I1130 02:39:33.576534       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 113 items received
I1130 02:39:33.591590       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=17354&timeout=8m8s&timeoutSeconds=488&watch=true 200 OK in 14 milliseconds
I1130 02:39:33.817636       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:39:33.817664       1 utils.go:79] GRPC request: {}
I1130 02:39:33.817724       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:39:42.504547       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:39:42.504626       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="19cf62af-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e52fe129-0651-45a3-8e6a-3ee546a82645" "latency"=61423696035
I1130 02:39:42.504662       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="19cf62af-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=61423846244
I1130 02:39:44.481451       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/
I1130 02:39:44.481508       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType 
I1130 02:39:44.481554       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e52fe129-0651-45a3-8e6a-3ee546a82645/globalmount with mount options([])
I1130 02:39:44.481577       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3])
... skipping 37 lines ...
I1130 02:39:45.268412       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:39:45.268428       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7a043c97-92cd-4c02-acbd-f4f681cf6136/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7a043c97-92cd-4c02-acbd-f4f681cf6136","csi.storage.k8s.io/pvc/name":"test.csi.azure.comgjw5m","csi.storage.k8s.io/pvc/namespace":"volumeio-1493","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-7a043c97-92cd-4c02-acbd-f4f681cf6136"}
I1130 02:39:45.268608       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7a043c97-92cd-4c02-acbd-f4f681cf6136-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:39:53.685116       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:39:53.685184       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:39:53.685218       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:39:53.685263       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:39:53.813631       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:39:57.283134       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:39:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="473aedbe-7058-11ed-bbee-000d3aad66aa"
I1130 02:39:57.291121       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 02:39:57.291723       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="473aedbe-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8633205
I1130 02:40:03.817866       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:40:03.817890       1 utils.go:79] GRPC request: {}
I1130 02:40:03.817938       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:40:07.807030       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:40:07.807103       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="4011a9ae-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7a043c97-92cd-4c02-acbd-f4f681cf6136" "latency"=22538417574
I1130 02:40:07.807130       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4011a9ae-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22538542081
I1130 02:40:08.471688       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1130 02:40:08.471749       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:40:08.471784       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7a043c97-92cd-4c02-acbd-f4f681cf6136/globalmount with mount options([])
I1130 02:40:08.471796       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 97 lines ...
I1130 02:41:04.005496       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-21731e1f-e8e7-44e8-b048-12e7d11e0484-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:41:21.665610       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 59 items received
I1130 02:41:21.674038       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=18241&timeout=8m8s&timeoutSeconds=488&watch=true 200 OK in 8 milliseconds
I1130 02:41:23.686260       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:41:23.686304       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:41:23.686347       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:41:23.686448       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:41:23.815739       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:41:27.284896       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:41:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="7ce01bb1-7058-11ed-bbee-000d3aad66aa"
I1130 02:41:27.293220       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 02:41:27.293375       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="7ce01bb1-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8545199
I1130 02:41:33.818156       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:41:33.818181       1 utils.go:79] GRPC request: {}
I1130 02:41:33.818233       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:41:45.298876       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:41:45.298956       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="6efff51e-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-21731e1f-e8e7-44e8-b048-12e7d11e0484" "latency"=41293362842
I1130 02:41:45.298992       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6efff51e-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=41293513752
I1130 02:41:45.592115       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:41:45.592139       1 utils.go:79] GRPC request: {}
I1130 02:41:45.592190       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:41:45.595514       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 56 lines ...
W1130 02:41:52.848451       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-21731e1f-e8e7-44e8-b048-12e7d11e0484/globalmount" is not a mountpoint, deleting
I1130 02:41:52.848540       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-21731e1f-e8e7-44e8-b048-12e7d11e0484/globalmount successfully
I1130 02:41:52.848553       1 utils.go:85] GRPC response: {}
I1130 02:41:53.687435       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:41:53.687441       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:41:53.687456       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:41:53.687576       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:41:53.816846       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:41:57.284068       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:41:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8ec19e3b-7058-11ed-bbee-000d3aad66aa"
I1130 02:41:57.291442       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 02:41:57.292200       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8ec19e3b-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8173178
I1130 02:42:03.817530       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:42:03.817557       1 utils.go:79] GRPC request: {}
I1130 02:42:03.817614       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:42:15.730029       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:42:15.730121       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="87ca5f75-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-81b48f0d-05f6-4c89-8ea5-9ffb9b3a3e7c" "latency"=30132672726
I1130 02:42:15.730165       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="87ca5f75-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30132819735
I1130 02:42:16.400345       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1130 02:42:16.400402       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:42:16.400434       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-81b48f0d-05f6-4c89-8ea5-9ffb9b3a3e7c/globalmount with mount options([])
I1130 02:42:16.400444       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 88 lines ...
I1130 02:43:47.761353       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:43:47.761370       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-454e2844-5dde-445f-a270-df3559660bcf/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-454e2844-5dde-445f-a270-df3559660bcf","csi.storage.k8s.io/pvc/name":"pvc-snapshottable-tester-hq88b-my-volume","csi.storage.k8s.io/pvc/namespace":"snapshotting-4875","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-454e2844-5dde-445f-a270-df3559660bcf"}
I1130 02:43:47.761537       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-454e2844-5dde-445f-a270-df3559660bcf-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:43:53.691333       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:43:53.691369       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:43:53.691425       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:43:53.691450       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:43:53.820836       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:43:56.728411       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:43:56.728437       1 utils.go:79] GRPC request: {}
I1130 02:43:56.728487       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:43:56.732864       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:43:56.732889       1 utils.go:79] GRPC request: {}
... skipping 4 lines ...
I1130 02:43:56.734703       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:43:56.734719       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26c23487-9c42-4c37-b697-8744e28d263f/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-26c23487-9c42-4c37-b697-8744e28d263f","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-w5cdt-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-7295","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-26c23487-9c42-4c37-b697-8744e28d263f"}
I1130 02:43:56.734890       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-26c23487-9c42-4c37-b697-8744e28d263f-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:43:57.283137       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:43:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d64805b3-7058-11ed-bbee-000d3aad66aa"
I1130 02:43:57.294602       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 11 milliseconds
I1130 02:43:57.294802       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d64805b3-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11706183
I1130 02:43:57.300855       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:43:57.300935       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d09b25cc-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-454e2844-5dde-445f-a270-df3559660bcf" "latency"=9539322267
I1130 02:43:57.300972       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d09b25cc-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9539445674
I1130 02:43:57.301286       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:43:57.301356       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d5f45f20-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-26c23487-9c42-4c37-b697-8744e28d263f" "latency"=566404775
I1130 02:43:57.301400       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d5f45f20-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=566529682
I1130 02:43:57.984669       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
I1130 02:43:57.984728       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 02:43:57.984761       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26c23487-9c42-4c37-b697-8744e28d263f/globalmount with mount options([])
I1130 02:43:57.984669       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
... skipping 93 lines ...
I1130 02:44:17.286676       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:44:17.286697       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-648b0537-5ce5-4de1-be5b-73aa5559bde2/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-648b0537-5ce5-4de1-be5b-73aa5559bde2","csi.storage.k8s.io/pvc/name":"restored-pvc-tester-bpx9d-my-volume","csi.storage.k8s.io/pvc/namespace":"snapshotting-4875","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-648b0537-5ce5-4de1-be5b-73aa5559bde2"}
I1130 02:44:17.287062       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-648b0537-5ce5-4de1-be5b-73aa5559bde2-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:44:23.692367       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:44:23.692412       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:44:23.692376       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:44:23.692512       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:44:23.822049       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:44:27.282250       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:44:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e82984a1-7058-11ed-bbee-000d3aad66aa"
I1130 02:44:27.295212       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 12 milliseconds
I1130 02:44:27.295458       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e82984a1-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13293976
I1130 02:44:33.817815       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:44:33.817847       1 utils.go:79] GRPC request: {}
... skipping 17 lines ...
W1130 02:44:34.713424       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26c23487-9c42-4c37-b697-8744e28d263f/globalmount" is not a mountpoint, deleting
I1130 02:44:34.713503       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26c23487-9c42-4c37-b697-8744e28d263f/globalmount successfully
I1130 02:44:34.713515       1 utils.go:85] GRPC response: {}
I1130 02:44:53.692888       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:44:53.692940       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:44:53.692975       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:44:53.692995       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:44:53.822221       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:44:57.284965       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:44:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="fa0b93bc-7058-11ed-bbee-000d3aad66aa"
I1130 02:44:57.297895       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 12 milliseconds
I1130 02:44:57.298069       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="fa0b93bc-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13138866
I1130 02:45:01.668854       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 139 items received
I1130 02:45:01.673480       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=19812&timeout=8m15s&timeoutSeconds=495&watch=true 200 OK in 4 milliseconds
I1130 02:45:03.818906       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:45:03.818930       1 utils.go:79] GRPC request: {}
I1130 02:45:03.818981       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:45:04.159869       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:45:04.159972       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e2345e11-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-648b0537-5ce5-4de1-be5b-73aa5559bde2" "latency"=46872782625
I1130 02:45:04.160015       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e2345e11-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=46873077942
I1130 02:45:04.869390       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1130 02:45:04.869443       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 02:45:04.869476       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-648b0537-5ce5-4de1-be5b-73aa5559bde2/globalmount with mount options([])
I1130 02:45:04.869488       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 36 lines ...
I1130 02:45:05.034117       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:45:05.034140       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b49818cc-e271-498d-bf87-b224954ee1bc","csi.storage.k8s.io/pvc/name":"test.csi.azure.comdmm8x","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-6831","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-b49818cc-e271-498d-bf87-b224954ee1bc"}
I1130 02:45:05.034319       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b49818cc-e271-498d-bf87-b224954ee1bc-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:45:23.692992       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:45:23.693068       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:45:23.693102       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:45:23.693148       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:45:23.822417       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:45:27.282579       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:45:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="0becd907-7059-11ed-bbee-000d3aad66aa"
I1130 02:45:27.289778       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 6 milliseconds
I1130 02:45:27.290057       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="0becd907-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7527940
I1130 02:45:29.785679       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:45:29.785752       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="feaa09d4-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b49818cc-e271-498d-bf87-b224954ee1bc" "latency"=24751353410
I1130 02:45:29.785783       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="feaa09d4-7058-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24751479218
I1130 02:45:30.456535       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1130 02:45:30.456593       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:45:30.456626       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/globalmount with mount options([])
I1130 02:45:30.456637       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 120 lines ...
I1130 02:45:49.610322       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/globalmount /var/lib/kubelet/pods/d373c726-93ed-4191-a4ea-6564774774fe/volumes/kubernetes.io~csi/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/mount)
I1130 02:45:49.611384       1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/globalmount at /var/lib/kubelet/pods/d373c726-93ed-4191-a4ea-6564774774fe/volumes/kubernetes.io~csi/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/mount successfully
I1130 02:45:49.611419       1 utils.go:85] GRPC response: {}
I1130 02:45:53.693990       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:45:53.694084       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:45:53.694082       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:45:53.694175       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:45:53.823509       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:45:55.294428       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1130 02:45:55.294457       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/pods/d373c726-93ed-4191-a4ea-6564774774fe/volumes/kubernetes.io~csi/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/mount","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-b49818cc-e271-498d-bf87-b224954ee1bc"}
I1130 02:45:55.294535       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-b49818cc-e271-498d-bf87-b224954ee1bc on /var/lib/kubelet/pods/d373c726-93ed-4191-a4ea-6564774774fe/volumes/kubernetes.io~csi/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/mount
I1130 02:45:55.294585       1 mount_helper_common.go:99] "/var/lib/kubelet/pods/d373c726-93ed-4191-a4ea-6564774774fe/volumes/kubernetes.io~csi/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/mount" is a mountpoint, unmounting
I1130 02:45:55.294602       1 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/d373c726-93ed-4191-a4ea-6564774774fe/volumes/kubernetes.io~csi/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/mount
... skipping 11 lines ...
W1130 02:45:55.410557       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/globalmount" is not a mountpoint, deleting
I1130 02:45:55.410627       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b49818cc-e271-498d-bf87-b224954ee1bc/globalmount successfully
I1130 02:45:55.410645       1 utils.go:85] GRPC response: {}
I1130 02:45:57.282531       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:45:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="1dce79da-7059-11ed-bbee-000d3aad66aa"
I1130 02:45:57.290634       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 02:45:57.290874       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="1dce79da-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8393892
I1130 02:46:03.210527       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:46:03.210604       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="18c9bc2d-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e731fe27-5b6c-45c9-b5b8-299a083b9c06" "latency"=14347716706
I1130 02:46:03.210648       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="18c9bc2d-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14347854214
I1130 02:46:03.817827       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:46:03.817850       1 utils.go:79] GRPC request: {}
I1130 02:46:03.817905       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:46:03.883165       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/
... skipping 68 lines ...
I1130 02:46:45.964381       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:46:45.965639       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:46:45.965668       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e731fe27-5b6c-45c9-b5b8-299a083b9c06/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e731fe27-5b6c-45c9-b5b8-299a083b9c06","csi.storage.k8s.io/pvc/name":"test.csi.azure.comqpvp7","csi.storage.k8s.io/pvc/namespace":"snapshotting-5488","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-e731fe27-5b6c-45c9-b5b8-299a083b9c06"}
I1130 02:46:45.966383       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e731fe27-5b6c-45c9-b5b8-299a083b9c06-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:46:53.695633       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:46:53.695673       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:46:53.695749       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:46:53.695782       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:46:53.825024       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:46:57.282992       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:46:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="4191d224-7059-11ed-bbee-000d3aad66aa"
I1130 02:46:57.297110       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 13 milliseconds
I1130 02:46:57.298393       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="4191d224-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15429501
I1130 02:47:03.818727       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:47:03.818752       1 utils.go:79] GRPC request: {}
I1130 02:47:03.818804       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:47:05.573749       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:47:05.573841       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="3ad2fc07-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e731fe27-5b6c-45c9-b5b8-299a083b9c06" "latency"=19607340953
I1130 02:47:05.573885       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3ad2fc07-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19607936687
I1130 02:47:05.902626       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:47:05.902655       1 utils.go:79] GRPC request: {}
I1130 02:47:05.902706       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:47:05.903670       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 52 lines ...
W1130 02:47:11.146541       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e731fe27-5b6c-45c9-b5b8-299a083b9c06/globalmount" is not a mountpoint, deleting
I1130 02:47:11.146637       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e731fe27-5b6c-45c9-b5b8-299a083b9c06/globalmount successfully
I1130 02:47:11.146660       1 utils.go:85] GRPC response: {}
I1130 02:47:23.696464       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:23.696631       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:23.696709       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:23.696745       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:47:23.826110       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:27.284759       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:47:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="5373ba59-7059-11ed-bbee-000d3aad66aa"
I1130 02:47:27.293216       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:47:27.294196       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="5373ba59-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9474754
I1130 02:47:33.819545       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:47:33.819575       1 utils.go:79] GRPC request: {}
I1130 02:47:33.819629       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:47:41.592466       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 89 items received
I1130 02:47:41.595207       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=21213&timeout=8m56s&timeoutSeconds=536&watch=true 200 OK in 2 milliseconds
I1130 02:47:53.697530       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:53.697657       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:47:53.697730       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:53.697772       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:53.827036       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:47:57.285174       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:47:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="65556c6f-7059-11ed-bbee-000d3aad66aa"
I1130 02:47:57.296690       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 11 milliseconds
I1130 02:47:57.296892       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="65556c6f-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11777488
I1130 02:48:03.818110       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:48:03.818139       1 utils.go:79] GRPC request: {}
I1130 02:48:03.818188       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:48:23.698313       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:23.698370       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:23.698401       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:23.698484       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:48:23.827824       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:27.285407       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:48:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="773719ec-7059-11ed-bbee-000d3aad66aa"
I1130 02:48:27.299002       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 13 milliseconds
I1130 02:48:27.299278       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="773719ec-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13887211
I1130 02:48:33.818357       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:48:33.818380       1 utils.go:79] GRPC request: {}
I1130 02:48:33.818426       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:48:53.698483       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:53.698570       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:53.698507       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:53.698755       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:48:53.828058       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:48:57.284677       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:48:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8918a089-7059-11ed-bbee-000d3aad66aa"
I1130 02:48:57.299559       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 14 milliseconds
I1130 02:48:57.299741       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8918a089-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15087681
I1130 02:49:03.817729       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:49:03.817759       1 utils.go:79] GRPC request: {}
I1130 02:49:03.817803       1 utils.go:85] GRPC response: {"ready":{"value":true}}
E1130 02:49:05.902917       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="46b56b09-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-105381c4-41bc-47a9-af3d-8bc3529d461c" "latency"=119997923881
E1130 02:49:05.902995       1 crdprovisioner.go:740]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="46b56b09-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119998145294
E1130 02:49:05.903021       1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-105381c4-41bc-47a9-af3d-8bc3529d461c) to be attached to node (k8s-agentpool1-10232909-vmss000000): context canceled
I1130 02:49:06.453833       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:49:06.453860       1 utils.go:79] GRPC request: {}
I1130 02:49:06.453909       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:49:06.454935       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:49:06.454956       1 utils.go:79] GRPC request: {}
I1130 02:49:06.455007       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:49:06.455951       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:49:06.455971       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-105381c4-41bc-47a9-af3d-8bc3529d461c","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-105381c4-41bc-47a9-af3d-8bc3529d461c","csi.storage.k8s.io/pvc/name":"pvc-hgqc6","csi.storage.k8s.io/pvc/namespace":"provisioning-7635","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-105381c4-41bc-47a9-af3d-8bc3529d461c"}
I1130 02:49:06.456173       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-105381c4-41bc-47a9-af3d-8bc3529d461c-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:49:23.699305       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:23.699313       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:23.699327       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:23.699517       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:49:23.828713       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:27.285359       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:49:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="9afa5d9d-7059-11ed-bbee-000d3aad66aa"
I1130 02:49:27.297526       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 11 milliseconds
I1130 02:49:27.297805       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="9afa5d9d-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12484630
I1130 02:49:29.674748       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 74 items received
I1130 02:49:29.678480       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=21829&timeout=9m45s&timeoutSeconds=585&watch=true 200 OK in 3 milliseconds
I1130 02:49:33.818442       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:49:33.818466       1 utils.go:79] GRPC request: {}
I1130 02:49:33.818518       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:49:41.951301       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:49:41.951401       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="8e9016cb-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-105381c4-41bc-47a9-af3d-8bc3529d461c" "latency"=35495140876
I1130 02:49:41.951444       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8e9016cb-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35495289684
I1130 02:49:42.283571       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:49:42.283600       1 utils.go:79] GRPC request: {}
I1130 02:49:42.283653       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:49:42.287366       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 30 lines ...
I1130 02:49:44.002266       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:49:44.002284       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-67588f4d-42f2-49fe-bff8-85afa533cb19","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-67588f4d-42f2-49fe-bff8-85afa533cb19","csi.storage.k8s.io/pvc/name":"pvc-4nl2k","csi.storage.k8s.io/pvc/namespace":"provisioning-7635","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-67588f4d-42f2-49fe-bff8-85afa533cb19"}
I1130 02:49:44.002470       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-67588f4d-42f2-49fe-bff8-85afa533cb19-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:49:53.700285       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:53.700319       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:53.700332       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:53.700410       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:49:53.700454       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:49:53.829634       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:49:55.445583       1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1130 02:49:55.445618       1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-105381c4-41bc-47a9-af3d-8bc3529d461c/5e1f666f-a3da-453d-abca-ef0fa1fb8cbc","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-105381c4-41bc-47a9-af3d-8bc3529d461c"}
I1130 02:49:55.445684       1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-105381c4-41bc-47a9-af3d-8bc3529d461c on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-105381c4-41bc-47a9-af3d-8bc3529d461c/5e1f666f-a3da-453d-abca-ef0fa1fb8cbc
I1130 02:49:55.445726       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-105381c4-41bc-47a9-af3d-8bc3529d461c/5e1f666f-a3da-453d-abca-ef0fa1fb8cbc" is a mountpoint, unmounting
I1130 02:49:55.445737       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-105381c4-41bc-47a9-af3d-8bc3529d461c/5e1f666f-a3da-453d-abca-ef0fa1fb8cbc
... skipping 14 lines ...
I1130 02:49:57.292480       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="acdb8de4-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10087790
I1130 02:50:03.818158       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:50:03.818182       1 utils.go:79] GRPC request: {}
I1130 02:50:03.818244       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:50:23.700417       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:23.700524       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:23.700549       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:50:23.700569       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:23.700579       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:50:23.829835       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:27.282395       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:50:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="bebd3075-7059-11ed-bbee-000d3aad66aa"
I1130 02:50:27.292733       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 10 milliseconds
I1130 02:50:27.293575       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="bebd3075-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11207654
I1130 02:50:33.818331       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:50:33.818358       1 utils.go:79] GRPC request: {}
I1130 02:50:33.818411       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:50:53.701426       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:53.701480       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:53.701499       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:53.701583       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:50:53.701609       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:50:53.830668       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:50:57.282320       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:50:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d09ecf83-7059-11ed-bbee-000d3aad66aa"
I1130 02:50:57.293824       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 11 milliseconds
I1130 02:50:57.294091       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="d09ecf83-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11820490
I1130 02:51:03.817749       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:51:03.817779       1 utils.go:79] GRPC request: {}
I1130 02:51:03.817825       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:51:07.879877       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:51:07.880041       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="a4f1334b-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-67588f4d-42f2-49fe-bff8-85afa533cb19" "latency"=83877414808
I1130 02:51:07.880083       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a4f1334b-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=83877618820
I1130 02:51:07.880331       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:51:07.880391       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="a3ebcf7b-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-aaa45a8e-bc06-43d8-9129-bcb04bab5749" "latency"=85590884907
I1130 02:51:07.880427       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a3ebcf7b-7059-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=85591012814
I1130 02:51:08.615858       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/
I1130 02:51:08.615909       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 02:51:08.615938       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-aaa45a8e-bc06-43d8-9129-bcb04bab5749/globalmount with mount options([])
I1130 02:51:08.615954       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 151 lines ...
I1130 02:53:38.574817       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:53:38.574835       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fe595920-93fe-4d9e-9e55-d502b0774cde/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-fe595920-93fe-4d9e-9e55-d502b0774cde","csi.storage.k8s.io/pvc/name":"test.csi.azure.comf248g","csi.storage.k8s.io/pvc/namespace":"provisioning-4200","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-fe595920-93fe-4d9e-9e55-d502b0774cde"}
I1130 02:53:38.575054       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-fe595920-93fe-4d9e-9e55-d502b0774cde-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:53:53.704630       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:53:53.704661       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:53:53.704692       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:53:53.704775       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:53:53.835016       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:53:57.282348       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:53:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="3be8a371-705a-11ed-bbee-000d3aad66aa"
I1130 02:53:57.291355       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 02:53:57.292073       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="3be8a371-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9755371
I1130 02:54:03.818338       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:54:03.818369       1 utils.go:79] GRPC request: {}
I1130 02:54:03.818419       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:54:17.373716       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:54:17.373788       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="30c22295-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fe595920-93fe-4d9e-9e55-d502b0774cde" "latency"=38798648931
I1130 02:54:17.373826       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="30c22295-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=38798791338
I1130 02:54:19.128195       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:54:19.128258       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:54:19.128314       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fe595920-93fe-4d9e-9e55-d502b0774cde/globalmount with mount options([])
I1130 02:54:19.128328       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 65 lines ...
I1130 02:54:49.754115       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:54:49.755051       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:54:49.755069       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c41273de-3852-42e6-bb6d-439f3d12e484/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c41273de-3852-42e6-bb6d-439f3d12e484","csi.storage.k8s.io/pvc/name":"test.csi.azure.com2stgq","csi.storage.k8s.io/pvc/namespace":"volume-5654","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-c41273de-3852-42e6-bb6d-439f3d12e484"}
I1130 02:54:49.755248       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c41273de-3852-42e6-bb6d-439f3d12e484-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:54:53.706639       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:54:53.706735       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:54:53.706794       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:54:53.706802       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:54:53.836271       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:54:57.285010       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:54:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="5fac5051-705a-11ed-bbee-000d3aad66aa"
I1130 02:54:57.295950       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 10 milliseconds
I1130 02:54:57.296540       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="5fac5051-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11583776
I1130 02:55:03.818236       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:55:03.818262       1 utils.go:79] GRPC request: {}
I1130 02:55:03.818319       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:55:23.707382       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:23.707416       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:23.707513       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:23.707593       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:55:23.836715       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:26.286209       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:55:26.286276       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="5b2f5f32-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c41273de-3852-42e6-bb6d-439f3d12e484" "latency"=36530947103
I1130 02:55:26.286308       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5b2f5f32-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=36531073411
I1130 02:55:27.282708       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:55:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="718d9a0c-705a-11ed-bbee-000d3aad66aa"
I1130 02:55:27.292417       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 9 milliseconds
I1130 02:55:27.292682       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="718d9a0c-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10002385
I1130 02:55:28.115887       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
... skipping 71 lines ...
I1130 02:55:33.818222       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:55:33.818249       1 utils.go:79] GRPC request: {}
I1130 02:55:33.818292       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:55:53.707969       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:53.708061       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:53.708101       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:53.708254       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:55:53.708330       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:55:53.837448       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:55:57.285098       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:55:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="836f9adf-705a-11ed-bbee-000d3aad66aa"
I1130 02:55:57.291628       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 6 milliseconds
I1130 02:55:57.291860       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="836f9adf-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6794097
I1130 02:55:59.736985       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:55:59.737056       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="7355841d-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-066c2102-bbc3-44ca-9559-520fefe0cd03" "latency"=29466388969
I1130 02:55:59.737078       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7355841d-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29466551378
I1130 02:55:59.741119       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:55:59.741186       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="7318453f-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-40a88bd6-f1bb-4698-a3b9-a2ccb63f24cb" "latency"=29871905160
I1130 02:55:59.741217       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7318453f-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29872060970
I1130 02:56:00.371808       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1130 02:56:00.371857       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType 
I1130 02:56:00.371901       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-40a88bd6-f1bb-4698-a3b9-a2ccb63f24cb/globalmount with mount options([])
I1130 02:56:00.371920       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2])
... skipping 125 lines ...
I1130 02:57:43.741248       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:57:43.741267       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-44f013e8-2009-4454-864d-5fa67849278c/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-44f013e8-2009-4454-864d-5fa67849278c","csi.storage.k8s.io/pvc/name":"pvc-7tr4n","csi.storage.k8s.io/pvc/namespace":"provisioning-1351","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-44f013e8-2009-4454-864d-5fa67849278c"}
I1130 02:57:43.741483       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-44f013e8-2009-4454-864d-5fa67849278c-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:57:53.711552       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:57:53.711583       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:57:53.711622       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:57:53.711651       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:57:53.841309       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:57:57.283716       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:57:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="caf5f028-705a-11ed-bbee-000d3aad66aa"
I1130 02:57:57.297550       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 13 milliseconds
I1130 02:57:57.297749       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="caf5f028-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=14085122
I1130 02:58:03.817535       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 02:58:03.817564       1 utils.go:79] GRPC request: {}
I1130 02:58:03.817618       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 02:58:18.937726       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:58:18.937795       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="c2e39058-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-44f013e8-2009-4454-864d-5fa67849278c" "latency"=35196211355
I1130 02:58:18.937822       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c2e39058-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35196360063
I1130 02:58:20.737403       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 02:58:20.737462       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 02:58:20.737494       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-44f013e8-2009-4454-864d-5fa67849278c/globalmount with mount options([])
I1130 02:58:20.737506       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 75 lines ...
I1130 02:59:15.575366       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 02:59:15.575395       1 utils.go:79] GRPC request: {}
I1130 02:59:15.575432       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 02:59:15.576279       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:59:15.576299       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8371cf85-cbda-43c4-a6d9-6044dca9c53c/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8371cf85-cbda-43c4-a6d9-6044dca9c53c","csi.storage.k8s.io/pvc/name":"pvc-c8hsf","csi.storage.k8s.io/pvc/namespace":"provisioning-1351","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-8371cf85-cbda-43c4-a6d9-6044dca9c53c"}
I1130 02:59:15.576513       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8371cf85-cbda-43c4-a6d9-6044dca9c53c-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:59:22.908509       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 02:59:22.908586       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="f9a079e1-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8371cf85-cbda-43c4-a6d9-6044dca9c53c" "latency"=7331983018
I1130 02:59:22.908633       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f9a079e1-705a-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7332136926
I1130 02:59:23.712578       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:59:23.712714       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:59:23.712778       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:59:23.843184       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 77 lines ...
I1130 02:59:49.417459       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 02:59:49.417480       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7d97a3fc-bf2a-4c9a-8f44-0df0a902998a/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7d97a3fc-bf2a-4c9a-8f44-0df0a902998a","csi.storage.k8s.io/pvc/name":"test.csi.azure.com7trb9","csi.storage.k8s.io/pvc/namespace":"multivolume-4696","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-7d97a3fc-bf2a-4c9a-8f44-0df0a902998a"}
I1130 02:59:49.417691       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7d97a3fc-bf2a-4c9a-8f44-0df0a902998a-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 02:59:53.713299       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:59:53.713387       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:59:53.713292       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:59:53.713456       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:59:53.713519       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 02:59:53.843902       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 02:59:57.282802       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T02:59:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="127c5870-705b-11ed-bbee-000d3aad66aa"
I1130 02:59:57.306356       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 23 milliseconds
I1130 02:59:57.306693       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="127c5870-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=23929297
I1130 03:00:03.818188       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:00:03.818221       1 utils.go:79] GRPC request: {}
I1130 03:00:03.818279       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:00:23.713390       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:00:23.713505       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:00:23.713554       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:00:23.713604       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:00:23.713633       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:00:23.844844       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:00:27.283206       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:00:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="245e0baa-705b-11ed-bbee-000d3aad66aa"
I1130 03:00:27.291266       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 7 milliseconds
I1130 03:00:27.291926       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="245e0baa-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8754110
I1130 03:00:27.956197       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:00:27.956269       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="0dc0291f-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1f66396d-3217-4913-94bb-2d0a1199e88b" "latency"=38617626092
I1130 03:00:27.956387       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0dc0291f-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=38617749999
I1130 03:00:27.957599       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:00:27.957657       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="0dcc3b05-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7d97a3fc-bf2a-4c9a-8f44-0df0a902998a" "latency"=38539910760
I1130 03:00:27.957688       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0dcc3b05-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=38540016467
I1130 03:00:28.697305       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1130 03:00:28.697349       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 03:00:28.697318       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/
I1130 03:00:28.697410       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1f66396d-3217-4913-94bb-2d0a1199e88b/globalmount with mount options([])
... skipping 71 lines ...
I1130 03:00:29.613416       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-413fcd81-86ef-4570-b732-d7979f7b8381-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 03:00:33.817556       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:00:33.817584       1 utils.go:79] GRPC request: {}
I1130 03:00:33.817638       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:00:41.693330       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 158 items received
I1130 03:00:41.712149       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=26781&timeout=6m56s&timeoutSeconds=416&watch=true 200 OK in 18 milliseconds
I1130 03:00:43.304699       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:00:43.304776       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="25c19cde-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-413fcd81-86ef-4570-b732-d7979f7b8381" "latency"=13691272673
I1130 03:00:43.304811       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="25c19cde-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13691414782
I1130 03:00:44.021929       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/
I1130 03:00:44.022013       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 03:00:44.022051       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-413fcd81-86ef-4570-b732-d7979f7b8381/globalmount with mount options([])
I1130 03:00:44.022063       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 174 lines ...
I1130 03:01:15.316123       1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-413fcd81-86ef-4570-b732-d7979f7b8381/globalmount
I1130 03:01:15.316160       1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-413fcd81-86ef-4570-b732-d7979f7b8381/globalmount" is a mountpoint, unmounting
I1130 03:01:15.316169       1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-413fcd81-86ef-4570-b732-d7979f7b8381/globalmount
W1130 03:01:15.333118       1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-413fcd81-86ef-4570-b732-d7979f7b8381/globalmount" is not a mountpoint, deleting
I1130 03:01:15.333196       1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-413fcd81-86ef-4570-b732-d7979f7b8381/globalmount successfully
I1130 03:01:15.333208       1 utils.go:85] GRPC response: {}
I1130 03:01:19.049791       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:01:19.049854       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="3c8015c7-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b7a9bbf5-547f-432b-870f-cc595e3cb25a" "latency"=11278222447
I1130 03:01:19.049886       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3c8015c7-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11278327153
I1130 03:01:19.054939       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:01:19.055012       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="3c801c8a-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e4897057-0b50-43ce-893a-455eba31ac78" "latency"=11283204438
I1130 03:01:19.055049       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3c801c8a-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11283313046
I1130 03:01:19.706853       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdc under /dev/disk/azure/scsi1/
I1130 03:01:19.706910       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType 
I1130 03:01:19.706949       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun4 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e4897057-0b50-43ce-893a-455eba31ac78/globalmount with mount options([nouuid])
I1130 03:01:19.706970       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun4" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun4])
... skipping 134 lines ...
I1130 03:02:28.264229       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:02:28.264251       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7fd1d3d6-2163-4c77-8fb1-eb92098426fa/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7fd1d3d6-2163-4c77-8fb1-eb92098426fa","csi.storage.k8s.io/pvc/name":"test.csi.azure.com5p6dh","csi.storage.k8s.io/pvc/namespace":"volume-2547","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-7fd1d3d6-2163-4c77-8fb1-eb92098426fa"}
I1130 03:02:28.264492       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7fd1d3d6-2163-4c77-8fb1-eb92098426fa-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 03:02:33.817955       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:02:33.818013       1 utils.go:79] GRPC request: {}
I1130 03:02:33.818063       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:02:35.861803       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:02:35.861898       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="6c7a5476-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7fd1d3d6-2163-4c77-8fb1-eb92098426fa" "latency"=7597315145
I1130 03:02:35.861943       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6c7a5476-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7597462254
I1130 03:02:36.556219       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/
I1130 03:02:36.556275       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 03:02:36.556309       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7fd1d3d6-2163-4c77-8fb1-eb92098426fa/globalmount with mount options([])
I1130 03:02:36.556336       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 142 lines ...
I1130 03:03:51.269139       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 03:03:51.269946       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:03:51.269991       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5b0d8921-50d8-4b7b-bebb-e659c05472c4/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5b0d8921-50d8-4b7b-bebb-e659c05472c4","csi.storage.k8s.io/pvc/name":"pvc-btjlt","csi.storage.k8s.io/pvc/namespace":"provisioning-176","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-5b0d8921-50d8-4b7b-bebb-e659c05472c4"}
I1130 03:03:51.270208       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5b0d8921-50d8-4b7b-bebb-e659c05472c4-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 03:03:53.718344       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:03:53.718507       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:03:53.718565       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:03:53.718539       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:03:53.851846       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:03:57.282982       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:03:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="a1897838-705b-11ed-bbee-000d3aad66aa"
I1130 03:03:57.297641       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 14 milliseconds
I1130 03:03:57.298421       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="a1897838-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=15467803
I1130 03:03:59.602660       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 67 items received
I1130 03:03:59.607344       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=28489&timeout=5m34s&timeoutSeconds=334&watch=true 200 OK in 4 milliseconds
I1130 03:04:03.819832       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:04:03.819867       1 utils.go:79] GRPC request: {}
I1130 03:04:03.819925       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:04:23.719711       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:23.719767       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:23.719791       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:23.719857       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:04:23.852048       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:27.284698       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:04:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="b36b5de3-705b-11ed-bbee-000d3aad66aa"
I1130 03:04:27.293584       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 03:04:27.294557       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="b36b5de3-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9882777
I1130 03:04:33.817712       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:04:33.817744       1 utils.go:79] GRPC request: {}
I1130 03:04:33.817813       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:04:53.722128       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:53.722159       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:53.722186       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:53.722277       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:04:53.852671       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:04:57.285355       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:04:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="c54d19b1-705b-11ed-bbee-000d3aad66aa"
I1130 03:04:57.298481       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 12 milliseconds
I1130 03:04:57.298722       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="c54d19b1-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13418884
I1130 03:05:03.817815       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:05:03.817843       1 utils.go:79] GRPC request: {}
I1130 03:05:03.817900       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:05:11.766518       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:05:11.766607       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="9df3fe6e-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5b0d8921-50d8-4b7b-bebb-e659c05472c4" "latency"=80496288047
I1130 03:05:11.766645       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9df3fe6e-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=80496469558
I1130 03:05:13.563214       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 03:05:13.563268       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 03:05:13.563299       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5b0d8921-50d8-4b7b-bebb-e659c05472c4/globalmount with mount options([])
I1130 03:05:13.563309       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 66 lines ...
I1130 03:05:52.250611       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:05:52.250630       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b36ef2a0-b424-4e4a-ac62-ba762a82768a/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b36ef2a0-b424-4e4a-ac62-ba762a82768a","csi.storage.k8s.io/pvc/name":"test.csi.azure.comhnqjj","csi.storage.k8s.io/pvc/namespace":"snapshotting-5743","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-b36ef2a0-b424-4e4a-ac62-ba762a82768a"}
I1130 03:05:52.250833       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b36ef2a0-b424-4e4a-ac62-ba762a82768a-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 03:05:53.724386       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:05:53.724448       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:05:53.724407       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:05:53.724587       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:05:53.853929       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:05:57.284715       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:05:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e9104751-705b-11ed-bbee-000d3aad66aa"
I1130 03:05:57.293819       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 8 milliseconds
I1130 03:05:57.294337       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e9104751-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9659865
I1130 03:06:03.819119       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:06:03.819169       1 utils.go:79] GRPC request: {}
I1130 03:06:03.819232       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:06:23.726015       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:23.726017       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:23.726030       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:23.726236       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:06:23.854586       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:06:27.285216       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:06:27Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="faf1fe2d-705b-11ed-bbee-000d3aad66aa"
I1130 03:06:27.295793       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 10 milliseconds
I1130 03:06:27.295986       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="faf1fe2d-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10811232
I1130 03:06:33.818315       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:06:33.818342       1 utils.go:79] GRPC request: {}
I1130 03:06:33.818393       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:06:51.975096       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:06:51.975184       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e6102cf8-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b36ef2a0-b424-4e4a-ac62-ba762a82768a" "latency"=59724245571
I1130 03:06:51.975250       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e6102cf8-705b-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59724424182
I1130 03:06:53.707370       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 03:06:53.707422       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 03:06:53.707467       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b36ef2a0-b424-4e4a-ac62-ba762a82768a/globalmount with mount options([])
I1130 03:06:53.707478       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 78 lines ...
I1130 03:07:35.877581       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 03:07:35.878435       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:07:35.878456       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-465e516a-653b-4b62-b389-37a2dc47bf00/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-465e516a-653b-4b62-b389-37a2dc47bf00","csi.storage.k8s.io/pvc/name":"pvc-vszrm","csi.storage.k8s.io/pvc/namespace":"snapshotting-5743","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-465e516a-653b-4b62-b389-37a2dc47bf00"}
I1130 03:07:35.878643       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-465e516a-653b-4b62-b389-37a2dc47bf00-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 03:07:37.712879       1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 113 items received
I1130 03:07:37.716486       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=29597&timeout=7m11s&timeoutSeconds=431&watch=true 200 OK in 3 milliseconds
I1130 03:07:47.174443       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:07:47.174518       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="23d48647-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-465e516a-653b-4b62-b389-37a2dc47bf00" "latency"=11295781804
I1130 03:07:47.174553       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="23d48647-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11295921512
I1130 03:07:48.855525       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 03:07:48.855586       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 03:07:48.855628       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-465e516a-653b-4b62-b389-37a2dc47bf00/globalmount with mount options([])
I1130 03:07:48.855641       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 80 lines ...
I1130 03:08:36.791385       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:08:36.791403       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-25f3d61e-caad-47c0-9504-9f30f5935069","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-25f3d61e-caad-47c0-9504-9f30f5935069","csi.storage.k8s.io/pvc/name":"pvc-n62mp","csi.storage.k8s.io/pvc/namespace":"provisioning-9376","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-25f3d61e-caad-47c0-9504-9f30f5935069"}
I1130 03:08:36.791582       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-25f3d61e-caad-47c0-9504-9f30f5935069-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 03:08:53.736146       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:08:53.736226       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:08:53.736225       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:08:53.736329       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:08:53.857658       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:08:57.284410       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:08:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="545a0cb8-705c-11ed-bbee-000d3aad66aa"
I1130 03:08:57.295380       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 10 milliseconds
I1130 03:08:57.295694       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="545a0cb8-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11324460
I1130 03:08:58.921902       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:08:58.922012       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="482319f8-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-25f3d61e-caad-47c0-9504-9f30f5935069" "latency"=22130335081
I1130 03:08:58.922084       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="482319f8-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22130506891
I1130 03:09:00.760955       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/
I1130 03:09:00.761009       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType 
I1130 03:09:00.761022       1 utils.go:85] GRPC response: {}
I1130 03:09:00.785248       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 47 lines ...
I1130 03:09:27.378538       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a710afdc-133a-4aff-b828-b43b65112a81-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 03:09:33.608267       1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 34 items received
I1130 03:09:33.615640       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=30366&timeout=8m7s&timeoutSeconds=487&watch=true 200 OK in 7 milliseconds
I1130 03:09:33.817700       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I1130 03:09:33.817726       1 utils.go:79] GRPC request: {}
I1130 03:09:33.817777       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I1130 03:09:43.929685       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:09:43.929756       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="664a0e57-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a710afdc-133a-4aff-b828-b43b65112a81" "latency"=16551124594
I1130 03:09:43.929784       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="664a0e57-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16551280604
I1130 03:09:45.700537       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 03:09:45.700595       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType 
I1130 03:09:45.700615       1 utils.go:85] GRPC response: {}
I1130 03:09:45.713346       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 71 lines ...
I1130 03:11:09.612037       1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I1130 03:11:09.612057       1 utils.go:79] GRPC request: {}
I1130 03:11:09.612090       1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I1130 03:11:09.612995       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:11:09.613015       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-31e37866-51ed-4844-b277-09d63006dc16/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-31e37866-51ed-4844-b277-09d63006dc16","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-31e37866-51ed-4844-b277-09d63006dc16"}
I1130 03:11:09.613210       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-31e37866-51ed-4844-b277-09d63006dc16-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 03:11:11.673678       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:11:11.673748       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="a339d476-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-31e37866-51ed-4844-b277-09d63006dc16" "latency"=2060472112
I1130 03:11:11.673774       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a339d476-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2060590819
I1130 03:11:13.442445       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1130 03:11:13.442513       1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS
I1130 03:11:13.442555       1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-31e37866-51ed-4844-b277-09d63006dc16/globalmount with mount options([])
I1130 03:11:13.442568       1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
... skipping 73 lines ...
I1130 03:12:42.023472       1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I1130 03:12:42.023511       1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-37233267-a1b5-42a6-88f7-8a98e0114867/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-37233267-a1b5-42a6-88f7-8a98e0114867","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-z8x6p-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1669774349551-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-5bbt3k3k/providers/Microsoft.Compute/disks/pvc-37233267-a1b5-42a6-88f7-8a98e0114867"}
I1130 03:12:42.023681       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-37233267-a1b5-42a6-88f7-8a98e0114867-k8s-agentpool1-10232909-vmss000000-attachment)
I1130 03:12:53.742430       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:12:53.742464       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:12:53.742510       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:12:53.742552       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I1130 03:12:53.861902       1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I1130 03:12:56.694467       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I1130 03:12:56.694549       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="da4e8ce5-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-37233267-a1b5-42a6-88f7-8a98e0114867" "latency"=14670783753
I1130 03:12:56.694584       1 crdprovisioner.go:740]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="da4e8ce5-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14670912261
I1130 03:12:57.283191       1 azuredisk_v2.go:556]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-30T03:12:57Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e366f4e7-705c-11ed-bbee-000d3aad66aa"
I1130 03:12:57.289665       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-10232909-vmss000000/status 200 OK in 6 milliseconds
I1130 03:12:57.289920       1 azuredisk_v2.go:543]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-10232909-vmss000000" "disk.csi.azure.com/request-id"="e366f4e7-705c-11ed-bbee-000d3aad66aa" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6792997
I1130 03:12:57.393258       1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/
... skipping 115 lines ...
I1130 02:12:19.137460       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?limit=500&resourceVersion=0 200 OK in 114 milliseconds
I1130 02:12:19.139787       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=2546&timeout=5m46s&timeoutSeconds=346&watch=true 200 OK in 1 milliseconds
I1130 02:12:19.221374       1 shared_informer.go:285] caches populated
I1130 02:12:19.221456       1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-ae2d55c668f6568106b7dd40ab11a7c3dbac74fb e2e-test
I1130 02:12:19.221469       1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider
I1130 02:12:19.224373       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I1130 02:12:19.224667       1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I1130 02:12:19.224684       1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider
I1130 02:12:19.224693       1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I1130 02:12:19.224725       1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully
I1130 02:12:19.225540       1 azure_auth.go:245] Using AzurePublicCloud environment
I1130 02:12:19.225562       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I1130 02:12:19.225570       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
... skipping 1742 lines ...
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-5bbt3k3k",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="300"} 215
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-5bbt3k3k",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="600"} 215
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-5bbt3k3k",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="1200"} 215
cloudprovider_azure_op_duration_seconds_bucket{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-5bbt3k3k",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e",le="+Inf"} 215
cloudprovider_azure_op_duration_seconds_sum{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-5bbt3k3k",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 7565.8602244730055
cloudprovider_azure_op_duration_seconds_count{request="azuredisk_csi_driver_controller_unpublish_volume",resource_group="kubetest-5bbt3k3k",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 215
# HELP cloudprovider_azure_op_failure_count [ALPHA] Number of failed Azure service operations
# TYPE cloudprovider_azure_op_failure_count counter
cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_controller_create_volume",resource_group="kubetest-5bbt3k3k",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 1
cloudprovider_azure_op_failure_count{request="azuredisk_csi_driver_controller_delete_volume",resource_group="kubetest-5bbt3k3k",source="test.csi.azure.com",subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e"} 6
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.0404e-05
... skipping 103 lines ...
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.7744128e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
make: *** [Makefile:341: e2e-test] Error 1
2022/11/30 03:14:28 process.go:155: Step 'make e2e-test' finished in 1h13m23.451035627s
2022/11/30 03:14:28 aksengine_helpers.go:425: downloading /root/tmp1439426005/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2022/11/30 03:14:28 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh
2022/11/30 03:14:28 process.go:153: Running: chmod +x /root/tmp1439426005/log-dump.sh
2022/11/30 03:14:28 process.go:155: Step 'chmod +x /root/tmp1439426005/log-dump.sh' finished in 2.115203ms
2022/11/30 03:14:28 aksengine_helpers.go:425: downloading /root/tmp1439426005/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml
... skipping 75 lines ...
ssh key file /root/.ssh/id_rsa does not exist. Exiting.
2022/11/30 03:15:29 process.go:155: Step 'bash -c /root/tmp1439426005/win-ci-logs-collector.sh kubetest-5bbt3k3k.westeurope.cloudapp.azure.com /root/tmp1439426005 /root/.ssh/id_rsa' finished in 4.088909ms
2022/11/30 03:15:29 aksengine.go:1141: Deleting resource group: kubetest-5bbt3k3k.
2022/11/30 03:22:55 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml.
2022/11/30 03:22:55 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"
2022/11/30 03:22:55 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 256.054324ms
2022/11/30 03:22:55 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2]
+ EXIT_VALUE=1
+ set +o xtrace
Cleaning up after docker in docker.
================================================================================
Cleaning up after docker
e5578303af32
... skipping 4 lines ...
PR	hccheng72: [V2] fix: delete orphaned azdrivernode and update error logs in Recover function
Result	FAILURE
Tests	1 failed / 13 succeeded
Started	2022-11-30 01:57
Elapsed	1h26m
Revision	d19baf154b9ae9b6cefe7e5f4ef58ad76b51b193
Refs	1615
job-version	v1.27.0-alpha.0.53+d98e9620e37995
kubetest-version	v20221116-7c85504268
revision	v1.27.0-alpha.0.53+d98e9620e37995
kubernetes-sigs/azuredisk-csi-driver PR #1615 pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1597771089977544704

Test Failures

kubetest Test 1h13m

Passed

Error lines from build-log.txt
