kubernetes-sigs/azuredisk-csi-driver PR #1570 pull-azuredisk-csi-driver-external-e2e-single-az-mainv2 #1590884429222383616
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | hccheng72: [V2] feat: watch CRD deletion to track driver uninstall |
| Result | FAILURE |
| Tests | 1 failed / 13 succeeded |
| Started | |
| Elapsed | 1h22m |
| Revision | 3b564021a6ed26e9b84e3489845c1621a19c3863 |
| Refs |
1570 |
| job-version | v1.26.0-beta.0.7+520e2c84b36a65 |
| kubetest-version | v20221109-489d560851 |
| revision | v1.26.0-beta.0.7+520e2c84b36a65 |
error during make e2e-test: exit status 2
from junit_runner.xml
Filter through log files | View test history on testgrid
kubetest Check APIReachability
kubetest Deferred TearDown
kubetest DumpClusterLogs
kubetest GetDeployer
kubetest IsUp
kubetest Prepare
kubetest TearDown
kubetest TearDown Previous
kubetest Timeout
kubetest Up
kubetest kubectl version
kubetest list nodes
kubetest test setup
... skipping 236 lines ... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 11156 100 11156 0 0 172k 0 --:--:-- --:--:-- --:--:-- 172k Downloading https://get.helm.sh/helm-v3.10.2-linux-amd64.tar.gz Verifying checksum... Done. Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 || make container-all push-manifest Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 not found: manifest unknown: manifest tagged by "latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=9ac0207e3b71fdd2858caea5dad8b15c528821d0 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2022-11-11T01:56:21Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin docker buildx rm container-builder || true ERROR: no builder "container-builder" found docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build # https://github.com/docker/buildx/issues/464#issuecomment-741507760 docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64 Unable to find image 'tonistiigi/binfmt:latest' locally ... skipping 705 lines ... } } ] } make[2]: Leaving directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 not found: manifest unknown: manifest tagged by "latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0" is not found make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver' docker buildx rm container-builder || true container-builder removed docker buildx create --use --name=container-builder container-builder # enable qemu for arm64 build ... skipping 1030 lines ... type: string type: object oneOf: - required: ["persistentVolumeClaimName"] - required: ["volumeSnapshotContentName"] volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 60 lines ... type: string volumeSnapshotContentName: description: volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent object representing an existing volume snapshot. This field should be set if the snapshot already exists and only needs a representation in Kubernetes. This field is immutable. type: string type: object volumeSnapshotClassName: description: 'VolumeSnapshotClassName is the name of the VolumeSnapshotClass requested by the VolumeSnapshot. VolumeSnapshotClassName may be left nil to indicate that the default SnapshotClass should be used. A given cluster may have multiple default Volume SnapshotClasses: one default per CSI Driver. If a VolumeSnapshot does not specify a SnapshotClass, VolumeSnapshotSource will be checked to figure out what the associated CSI Driver is, and the default VolumeSnapshotClass associated with that CSI Driver will be used. If more than one VolumeSnapshotClass exist for a given CSI Driver and more than one have been marked as default, CreateSnapshot will fail and generate an event. Empty string is not allowed for this field.' type: string required: - source type: object status: description: status represents the current information of a snapshot. Consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object. ... skipping 2 lines ... description: 'boundVolumeSnapshotContentName is the name of the VolumeSnapshotContent object to which this VolumeSnapshot object intends to bind to. If not specified, it indicates that the VolumeSnapshot object has not been successfully bound to a VolumeSnapshotContent object yet. NOTE: To avoid possible security issues, consumers must verify binding between VolumeSnapshot and VolumeSnapshotContent objects is successful (by validating that both VolumeSnapshot and VolumeSnapshotContent point at each other) before using this object.' type: string creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it may indicate that the creation time of the snapshot is unknown. format: date-time type: string error: description: error is the last observed error during snapshot creation, if any. This field could be helpful to upper level controllers(i.e., application controller) to decide whether they should continue on waiting for the snapshot to be created based on the type of error reported. The snapshot controller will keep retrying when an error occurs during the snapshot creation. Upon success, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if the snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: type: string description: restoreSize represents the minimum size of volume required to create a volume from this snapshot. In dynamic snapshot creation case, this field will be filled in by the snapshot controller with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ x-kubernetes-int-or-string: true type: object required: - spec type: object ... skipping 254 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 108 lines ... description: status represents the current information of a snapshot. properties: creationTime: description: creationTime is the timestamp when the point-in-time snapshot is taken by the underlying storage system. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "creation_time" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "creation_time" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. If not specified, it indicates the creation time is unknown. The format of this field is a Unix nanoseconds time encoded as an int64. On Unix, the command `date +%s%N` returns the current time in nanoseconds since 1970-01-01 00:00:00 UTC. format: int64 type: integer error: description: error is the last observed error during snapshot creation, if any. Upon success after retry, this error field will be cleared. properties: message: description: 'message is a string detailing the encountered error during snapshot creation if specified. NOTE: message may be logged, and it should not contain sensitive information.' type: string time: description: time is the timestamp when the error was encountered. format: date-time type: string type: object readyToUse: description: readyToUse indicates if a snapshot is ready to be used to restore a volume. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "ready_to_use" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "ready_to_use" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it, otherwise, this field will be set to "True". If not specified, it means the readiness of a snapshot is unknown. type: boolean restoreSize: description: restoreSize represents the complete size of the snapshot in bytes. In dynamic snapshot creation case, this field will be filled in by the CSI snapshotter sidecar with the "size_bytes" value returned from CSI "CreateSnapshot" gRPC call. For a pre-existing snapshot, this field will be filled with the "size_bytes" value returned from the CSI "ListSnapshots" gRPC call if the driver supports it. When restoring a volume from this snapshot, the size of the volume MUST NOT be smaller than the restoreSize if it is specified, otherwise the restoration will fail. If not specified, it indicates that the size is unknown. format: int64 minimum: 0 type: integer snapshotHandle: description: snapshotHandle is the CSI "snapshot_id" of a snapshot on the underlying storage system. If not specified, it indicates that dynamic snapshot creation has either failed or it is still in progress. type: string type: object required: - spec type: object served: true ... skipping 359 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status Required properties: detail: description: Status summarizes the current attachment state of the volume attachment Nil Status indicates that the volume has not yet been attached to the node properties: ... skipping 7 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 90 lines ... - volumeName - volume_context - volume_id type: object status: description: status represents the current state of AzVolumeAttachment. includes error, state, and attachment status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 13 lines ... role: description: The current attachment role. type: string required: - role type: object error: description: Error occurred during attach/detach of volume properties: code: type: string message: type: string parameters: ... skipping 169 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: detail: description: Current status detail of the AzVolume Nil detail indicates that the volume has not been created properties: accessible_topology: ... skipping 28 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 154 lines ... - maxMountReplicaCount - volumeCapability - volumeName type: object status: description: status represents the current state of AzVolume. includes error, state, and volume status properties: annotation: additionalProperties: type: string description: Annotations contains additional resource information to guide driver actions ... skipping 34 lines ... type: string required: - capacity_bytes - node_expansion_required - volume_id type: object error: description: Error occurred during creation/deletion of volume properties: code: type: string message: type: string parameters: ... skipping 1069 lines ... image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0" args: - "-csi-address=$(ADDRESS)" - "-v=2" - "-leader-election" - "--leader-election-namespace=kube-system" - '-handle-volume-inuse-error=false' - '-feature-gates=RecoverVolumeExpansionFailure=true' - "-timeout=240s" env: - name: ADDRESS value: /csi/csi.sock volumeMounts: ... skipping 323 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:269[0m [36mDistro debian doesn't support ntfs -- skipping[0m test/e2e/storage/framework/testsuite.go:127 [90m------------------------------[0m ... skipping 81 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:258[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 69 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:280[0m [36mDistro debian doesn't support ntfs -- skipping[0m test/e2e/storage/framework/testsuite.go:127 [90m------------------------------[0m ... skipping 313 lines ... Nov 11 02:12:01.575: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m36.048010167s) Nov 11 02:12:06.683: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m41.156643958s) Nov 11 02:12:11.795: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m46.26857884s) Nov 11 02:12:16.911: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m51.384404203s) Nov 11 02:12:22.029: INFO: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 found and phase=Released (4m56.502878528s) [1mSTEP[0m: Deleting sc Nov 11 02:12:27.141: FAIL: while cleaning up resource Unexpected error: <errors.aggregate | len:1, cap:1>: [ [ { msg: "persistent Volume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 not deleted by dynamic provisioner: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s", err: { s: "PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s", ... skipping 33 lines ... Nov 11 02:12:27.253: INFO: At 2022-11-11 02:07:14 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {default-scheduler } Scheduled: Successfully assigned volumemode-3690/hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc to k8s-agentpool1-30725374-vmss000001 Nov 11 02:12:27.253: INFO: At 2022-11-11 02:07:18 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {kubelet k8s-agentpool1-30725374-vmss000001} Pulled: Successfully pulled image "k8s.gcr.io/e2e-test-images/agnhost:2.36" in 3.794195202s Nov 11 02:12:27.253: INFO: At 2022-11-11 02:07:18 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {kubelet k8s-agentpool1-30725374-vmss000001} Started: Started container agnhost-container Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:18 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {kubelet k8s-agentpool1-30725374-vmss000001} Created: Created container agnhost-container Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:23 +0000 UTC - event for hostexec-k8s-agentpool1-30725374-vmss000001-bl4tc: {kubelet k8s-agentpool1-30725374-vmss000001} Killing: Stopping container agnhost-container Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:23 +0000 UTC - event for pod-3d455c00-a9bd-466e-b16b-83b2965556cd: {kubelet k8s-agentpool1-30725374-vmss000001} Killing: Stopping container write-pod Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:25 +0000 UTC - event for pod-3d455c00-a9bd-466e-b16b-83b2965556cd: {kubelet k8s-agentpool1-30725374-vmss000001} FailedKillPod: error killing pod: failed to "KillContainer" for "write-pod" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: No such container: d51e545125c0659d71cc059b1389f9cd7db490e82ff391d2861ec0cdcf0151a6" Nov 11 02:12:27.254: INFO: At 2022-11-11 02:07:36 +0000 UTC - event for pod-3d455c00-a9bd-466e-b16b-83b2965556cd: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025" Nov 11 02:12:27.363: INFO: POD NODE PHASE GRACE CONDITIONS Nov 11 02:12:27.363: INFO: Nov 11 02:12:27.579: INFO: Logging node info for node k8s-agentpool1-30725374-vmss000000 Nov 11 02:12:27.689: INFO: Node Info: &Node{ObjectMeta:{k8s-agentpool1-30725374-vmss000000 9e7f2eeb-24ce-481f-8f7f-b3379d8377a6 4487 0 2022-11-11 01:54:07 +0000 UTC <nil> <nil> map[agentpool:agentpool1 beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:Standard_D8s_v3 beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:westeurope failure-domain.beta.kubernetes.io/zone:0 kubernetes.azure.com/cluster:kubetest-9juvoppw kubernetes.azure.com/role:agent kubernetes.io/arch:amd64 kubernetes.io/hostname:k8s-agentpool1-30725374-vmss000000 kubernetes.io/os:linux kubernetes.io/role:agent node-role.kubernetes.io/agent: node.kubernetes.io/instance-type:Standard_D8s_v3 storageprofile:managed storagetier:Premium_LRS topology.kubernetes.io/region:westeurope topology.kubernetes.io/zone:0 topology.test.csi.azure.com/zone:] map[csi.volume.kubernetes.io/nodeid:{"test.csi.azure.com":"k8s-agentpool1-30725374-vmss000000"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2022-11-11 01:54:07 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:agentpool":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/os":{},"f:kubernetes.azure.com/cluster":{},"f:kubernetes.azure.com/role":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:storageprofile":{},"f:storagetier":{}}}} } {cloud-controller-manager Update v1 2022-11-11 01:54:22 +0000 UTC FieldsV1 {"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.244.3.0/24\"":{}}}} } {kubectl-label Update v1 2022-11-11 01:54:23 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/agent":{}}}} } {kube-controller-manager Update v1 2022-11-11 01:54:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}}} } {cloud-controller-manager Update v1 2022-11-11 01:55:09 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {cloud-node-manager Update v1 2022-11-11 01:55:46 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:beta.kubernetes.io/instance-type":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kubelet Update v1 2022-11-11 02:07:12 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.test.csi.azure.com/zone":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{},"f:volumesInUse":{}}} status} {kube-controller-manager Update v1 2022-11-11 02:07:35 +0000 UTC FieldsV1 {"f:status":{"f:volumesAttached":{}}} status}]},Spec:NodeSpec{PodCIDR:10.244.3.0/24,DoNotUseExternalID:,ProviderID:azure:///subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/virtualMachineScaleSets/k8s-agentpool1-30725374-vmss/virtualMachines/0,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.244.3.0/24],},Status:NodeStatus{Capacity:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{31025332224 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{33672720384 0} {<nil>} 32883516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{cpu: {{8 0} {<nil>} 8 DecimalSI},ephemeral-storage: {{27922798956 0} {<nil>} 27922798956 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{32886288384 0} {<nil>} 32115516Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-11-11 01:55:09 +0000 UTC,LastTransitionTime:2022-11-11 01:55:09 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-11-11 02:12:07 +0000 UTC,LastTransitionTime:2022-11-11 01:54:03 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-11-11 02:12:07 +0000 UTC,LastTransitionTime:2022-11-11 01:54:03 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-11-11 02:12:07 +0000 UTC,LastTransitionTime:2022-11-11 01:54:03 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-11-11 02:12:07 +0000 UTC,LastTransitionTime:2022-11-11 01:54:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.240.0.4,},NodeAddress{Type:Hostname,Address:k8s-agentpool1-30725374-vmss000000,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:9dc7951faf284e71b31f4fbcc187bf1c,SystemUUID:04cd8ce5-2ce6-a04f-bb6b-13202e75538b,BootID:95f20e2d-1e4e-43d6-a374-9c013de158b4,KernelVersion:5.4.0-1094-azure,OSImage:Ubuntu 18.04.6 LTS,ContainerRuntimeVersion:docker://20.10.14+azure-1,KubeletVersion:v1.23.13,KubeProxyVersion:v1.23.13,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi@sha256:423eb6cf602c064c8b2deefead5ceadd6324ed41b3d995dab5d0f6f0f4d4710f mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.10.0],SizeBytes:245959792,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi@sha256:9e2ecabcf9dd9943e6600eb9fb460f45b4dc61af7cabe95d115082a029db2aaf mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.9.0],SizeBytes:230470852,},ContainerImage{Names:[mcr.microsoft.com/containernetworking/azure-npm@sha256:106f669f48e5e80c4ec0afb49858ead72cf4b901cd8664e7bf81f8d789e56e12 mcr.microsoft.com/containernetworking/azure-npm:v1.2.2_hotfix],SizeBytes:175230380,},ContainerImage{Names:[k8sprow.azurecr.io/azuredisk-csi@sha256:cbf9a288cfc596f962ae3401e7610f25d37321c217da7323c62a28d746eedea9 k8sprow.azurecr.io/azuredisk-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0],SizeBytes:142917021,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-apiserver@sha256:46c1541fe63e34dfc697e3acbf8d40661ba8af217bb5457b8ff5adb9af4b248e mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.23.13],SizeBytes:135172310,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-controller-manager@sha256:041c9d6da46acbdbaaa937ef2f370a341cf86c77654304ce3b04ab82f28b4952 mcr.microsoft.com/oss/kubernetes/kube-controller-manager:v1.23.13],SizeBytes:124990264,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver@sha256:c0d040a1c4fbfceb65663e31c09ea40f4f78e356437610cbc3fbb4bb409bd6f1 mcr.microsoft.com/oss/kubernetes-csi/secrets-store/driver:v0.0.19],SizeBytes:123229697,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-proxy@sha256:9aeab889b5d0623cd5ca7aceb506ed2bc34836ee99fec603bfd0e0abe95e983e mcr.microsoft.com/oss/kubernetes/kube-proxy:v1.23.13],SizeBytes:112304283,},ContainerImage{Names:[mcr.microsoft.com/oss/azure/secrets-store/provider-azure@sha256:6f67f3d0c7cdde5702f8ce7f101b6519daa0237f0c34fecb7c058b6af8c22ad1 mcr.microsoft.com/oss/azure/secrets-store/provider-azure:0.0.12],SizeBytes:101061355,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler@sha256:6f0c680d375c62e74351f8ff3ed6ddb9b72ca759e0645c329b95f64264654a6d mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-autoscaler:v1.22.1],SizeBytes:99962810,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:32e2836018c96e73533bd4642fe438e465b81dcbfa8b7b61935a6f4d0246c7ae mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.3],SizeBytes:86832059,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-addon-manager@sha256:92c2c5aad9012ee32d2a43a74966cc0adc6ccb1705ad15abb10485ecf406d88b mcr.microsoft.com/oss/kubernetes/kube-addon-manager:v9.1.5],SizeBytes:84094027,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/metrics-server@sha256:1ef9d57ce41ffcc328b92494c3bfafe401e0b9a1694a295301a1385337d52815 mcr.microsoft.com/oss/kubernetes/metrics-server:v0.5.2],SizeBytes:64327621,},ContainerImage{Names:[mcr.microsoft.com/oss/nvidia/k8s-device-plugin@sha256:0f5b52bf28239234e831697d96db63ac03cde70fe68058f964504ab7564ee810 mcr.microsoft.com/oss/nvidia/k8s-device-plugin:1.0.0-beta6],SizeBytes:64160241,},ContainerImage{Names:[k8sprow.azurecr.io/azdiskschedulerextender-csi@sha256:3ed520c42107e2374e4d039698ce45f2c7891721708c2d9534dc3a87181939af k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0],SizeBytes:58129683,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner@sha256:e9ddadc44ba87a4a27f67e54760a14f9986885b534b3dff170a14eae1e35d213 mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.0.0],SizeBytes:56881280,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-resizer@sha256:c5bb71ceaac60b1a4b58739fa07b709f6248c452ff6272a384d2f7648895a750 mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.3.0],SizeBytes:54313772,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter@sha256:61849a026511cf332c87d73d0a7aed803b510c3ede197ec755389686d490de72 mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v4.2.1],SizeBytes:54210936,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-attacher@sha256:6b41e7153ebdfdc1501aa65184624bc15fd33a52d93f88ec3a758d0f8c9b8c10 mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.3.0],SizeBytes:53842561,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:95d70317663e1da01ef37d42cce2e35241292b777e934ec4e58089e3d69a7e10 mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.23.13],SizeBytes:53486364,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:8c3fc3c2667004ad6bbdf723bb64c5da66a5cb8b11d4ee59b67179b686223b13 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v5.0.1],SizeBytes:52732401,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller@sha256:be5a8dc1f990828f653c77bd0a0f1bbd13197c3019f6e1d99d590389bac36705 mcr.microsoft.com/oss/kubernetes-csi/snapshot-controller:v4.2.1],SizeBytes:51575245,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:949db4eea46ea31f9afa0b49ad0b55f7a31f91aa03b408e0ac9b7720c8ef7d29 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.23.11],SizeBytes:51376235,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:72fbc9d17947d1a54c09a16671e6810fd7c0d8f09c0502fdbdd212c2192451a7 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.1.14],SizeBytes:51044459,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:075ea1f8270312350f1396ab6677251e803e61a523822d5abfa5e6acd180cfab mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.23.11],SizeBytes:50806891,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/kube-scheduler@sha256:c35f45e8dfaa30363e589296415a057f9d32813920fc3820d6557491657f199a mcr.microsoft.com/oss/kubernetes/kube-scheduler:v1.21.2],SizeBytes:50607491,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/ip-masq-agent@sha256:1244155f2ed3f33ff154cc343b8ad285f3391d95afd7d4b1c6dcc420bc0ba3cf mcr.microsoft.com/oss/kubernetes/ip-masq-agent:v2.5.0],SizeBytes:50146762,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:eb7d329559d954093f7a234bc0ff322764de48a0b2a1007229032fb53e7bb9eb mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.24.0],SizeBytes:49070187,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:2dfff4dc10f6d25b0a34c01bf4d166540cbf1ff2bccd52635350039138eb23c4 mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.0.18],SizeBytes:48775275,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager@sha256:3e23b9ab6bfee42a1751019c21a60aa474964bd4cba00cbed5055430424242cc mcr.microsoft.com/oss/kubernetes/azure-cloud-node-manager:v1.24.0],SizeBytes:48500843,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:acec9ee6626c2cdea89107ba191ae36e6f60339c9b3b3215b5a59bf0e2e730ba mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.1.14],SizeBytes:47853675,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/coredns@sha256:f873bf7f0928461efe10697fa76cf0ad7a1ae3041c5b57b50dd3d0b72d273f8c mcr.microsoft.com/oss/kubernetes/coredns:1.8.6],SizeBytes:46804601,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager@sha256:759ac7020e982884b99121583bd577658b5663b1c1ed69746356aa6ce9879da4 mcr.microsoft.com/oss/kubernetes/azure-cloud-controller-manager:v1.0.18],SizeBytes:45850731,},ContainerImage{Names:[mcr.microsoft.com/oss/etcd-io/etcd@sha256:cf587862e3f1b6fa4d9a2565520a34f164bdf72c50f37af8c3c668160593246e mcr.microsoft.com/oss/etcd-io/etcd:v3.3.25],SizeBytes:41832119,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/mic@sha256:bd9465be94966b9a917e1e3904fa5e63dd91772ccadf304e18ffd8e4ad8ccedd mcr.microsoft.com/k8s/aad-pod-identity/mic:1.6.1],SizeBytes:41374894,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler@sha256:c849d75d61943ce7f51b4c049f1a79d19a08253966c8f49c4cfb6414cc33db8b mcr.microsoft.com/oss/kubernetes/autoscaler/cluster-proportional-autoscaler:1.8.5],SizeBytes:40661903,},ContainerImage{Names:[mcr.microsoft.com/k8s/aad-pod-identity/nmi@sha256:02128fefcdb7593ac53fc342e2c53a0fc6fabd813036bf60457bf43cc2940116 mcr.microsoft.com/k8s/aad-pod-identity/nmi:1.6.1],SizeBytes:38007982,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume@sha256:921f301c44dda06a325164accf22e78ecc570b5c7d9d6ee4c66bd8cbb2b60b9a mcr.microsoft.com/k8s/flexvolume/keyvault-flexvolume:v0.0.16],SizeBytes:26970670,},ContainerImage{Names:[mcr.microsoft.com/k8s/kms/keyvault@sha256:1a27e175f8c125209e32d2957b5509fe20757bd8cb309ff9da598799b56326fb mcr.microsoft.com/k8s/kms/keyvault:v0.0.10],SizeBytes:23077387,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:2fbd1e1a0538a06f2061afd45975df70c942654aa7f86e920720169ee439c2d6 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1],SizeBytes:19743433,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar@sha256:dbec3a8166686b09b242176ab5b99e993da4126438bbce68147c3fd654f35662 mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.4.0],SizeBytes:19547289,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:31547791294872570393470991481c2477a311031d3a03e0ae54eb164347dc34 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0],SizeBytes:17857485,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes-csi/livenessprobe@sha256:c96a6255c42766f6b8bb1a7cda02b0060ab1b20b2e2dafcc64ec09e7646745a6 mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.5.0],SizeBytes:17573341,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:13616070e3f29de4417eee434a8ef472221c9e51b3d037b5a6b46cef08eb7443 k8s.gcr.io/e2e-test-images/nginx:1.14-2],SizeBytes:16028126,},ContainerImage{Names:[mcr.microsoft.com/oss/busybox/busybox@sha256:582a641242b49809af3a1a522f9aae8c3f047d1c6ca1dd9d8cdabd349e45b1a9 mcr.microsoft.com/oss/busybox/busybox:1.33.1],SizeBytes:1235829,},ContainerImage{Names:[mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume@sha256:23d8c6033f02a1ecad05127ebdc931bb871264228661bc122704b0974e4d9fdd mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.8],SizeBytes:1159025,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:c318242786b139d18676b1c09a0ad7f15fc17f8f16a5b2e625cd0dc8c9703daf k8s.gcr.io/e2e-test-images/busybox:1.29-2],SizeBytes:1154361,},ContainerImage{Names:[mcr.microsoft.com/oss/kubernetes/pause@sha256:e3b8c20681593c21b344ad801fbb8abaf564427ee3a57a9fcfa3b455f917ce46 mcr.microsoft.com/oss/kubernetes/pause:3.4.1],SizeBytes:682696,},},VolumesInUse:[kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-75b7eb16-398c-42f1-994f-7aa02a0ccdd8 kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-d6226a7d-bdbf-4b17-a53a-8f979cb0ca11],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-75b7eb16-398c-42f1-994f-7aa02a0ccdd8,DevicePath:,},AttachedVolume{Name:kubernetes.io/csi/test.csi.azure.com^/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-d6226a7d-bdbf-4b17-a53a-8f979cb0ca11,DevicePath:,},},Config:nil,},} ... skipping 114 lines ... [Testpattern: Dynamic PV (block volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould not mount / map unused volumes in a pod [LinuxOnly] [Measurement][0m [90mtest/e2e/storage/testsuites/volumemode.go:354[0m [91mNov 11 02:12:27.141: while cleaning up resource Unexpected error: <errors.aggregate | len:1, cap:1>: [ [ { msg: "persistent Volume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 not deleted by dynamic provisioner: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s", err: { s: "PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s", ... skipping 3 lines ... ] persistent Volume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 not deleted by dynamic provisioner: PersistentVolume pvc-85c099dd-62fb-4b3b-becb-d08abd7aa025 still exists within 5m0s occurred[0m test/e2e/storage/testsuites/volumemode.go:190 [90m------------------------------[0m {"msg":"FAILED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":31,"completed":0,"skipped":7,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:12:30.828: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.002 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Inline-volume (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:280[0m [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 44 lines ... Nov 11 02:07:05.418: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comfdlbg] to have phase Bound Nov 11 02:07:05.526: INFO: PersistentVolumeClaim test.csi.azure.comfdlbg found but phase is Pending instead of Bound. Nov 11 02:07:07.635: INFO: PersistentVolumeClaim test.csi.azure.comfdlbg found but phase is Pending instead of Bound. Nov 11 02:07:09.743: INFO: PersistentVolumeClaim test.csi.azure.comfdlbg found and phase=Bound (4.324617683s) [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-pp4s [1mSTEP[0m: Creating a pod to test exec-volume-test Nov 11 02:07:10.084: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-pp4s" in namespace "volume-154" to be "Succeeded or Failed" Nov 11 02:07:10.192: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 107.768391ms Nov 11 02:07:12.301: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216969353s Nov 11 02:07:14.410: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325914972s Nov 11 02:07:16.521: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436256443s Nov 11 02:07:18.629: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 8.544873882s Nov 11 02:07:20.742: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657486413s ... skipping 122 lines ... Nov 11 02:11:40.236: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.151496323s Nov 11 02:11:42.346: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.261111463s Nov 11 02:11:44.455: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.370372229s Nov 11 02:11:46.565: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4m36.480825034s Nov 11 02:11:48.675: INFO: Pod "exec-volume-test-dynamicpv-pp4s": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4m38.590370543s [1mSTEP[0m: Saw pod success Nov 11 02:11:48.675: INFO: Pod "exec-volume-test-dynamicpv-pp4s" satisfied condition "Succeeded or Failed" Nov 11 02:11:48.790: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod exec-volume-test-dynamicpv-pp4s container exec-container-dynamicpv-pp4s: <nil> [1mSTEP[0m: delete the pod Nov 11 02:11:49.049: INFO: Waiting for pod exec-volume-test-dynamicpv-pp4s to disappear Nov 11 02:11:49.157: INFO: Pod exec-volume-test-dynamicpv-pp4s no longer exists [1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-pp4s Nov 11 02:11:49.157: INFO: Deleting pod "exec-volume-test-dynamicpv-pp4s" in namespace "volume-154" ... skipping 23 lines ... [90mtest/e2e/storage/framework/testsuite.go:50[0m should allow exec of files on the volume [90mtest/e2e/storage/testsuites/volumes.go:198[0m [90m------------------------------[0m [36mS[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should allow exec of files on the volume","total":27,"completed":1,"skipped":15,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] volumeIO test/e2e/storage/framework/testsuite.go:51 Nov 11 02:12:30.914: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 232 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:138[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":28,"completed":1,"skipped":85,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy[0m [1m(OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents[0m [37mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m ... skipping 119 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents","total":38,"completed":1,"skipped":291,"failed":0} [36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:13:22.271: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 300 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:138[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":41,"completed":1,"skipped":200,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Nov 11 02:13:50.066: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 104 lines ... Nov 11 02:07:05.490: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com7t57r] to have phase Bound Nov 11 02:07:05.598: INFO: PersistentVolumeClaim test.csi.azure.com7t57r found but phase is Pending instead of Bound. Nov 11 02:07:07.707: INFO: PersistentVolumeClaim test.csi.azure.com7t57r found but phase is Pending instead of Bound. Nov 11 02:07:09.815: INFO: PersistentVolumeClaim test.csi.azure.com7t57r found and phase=Bound (4.324955938s) [1mSTEP[0m: [init] starting a pod to use the claim [1mSTEP[0m: [init] check pod success Nov 11 02:07:10.259: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-7jn44" in namespace "snapshotting-4409" to be "Succeeded or Failed" Nov 11 02:07:10.368: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 108.238798ms Nov 11 02:07:12.478: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218518842s Nov 11 02:07:14.587: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327111958s Nov 11 02:07:16.696: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436368303s Nov 11 02:07:18.809: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 8.5495259s Nov 11 02:07:20.919: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 10.659881919s ... skipping 121 lines ... Nov 11 02:11:38.329: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4m28.069904174s Nov 11 02:11:40.440: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4m30.180749938s Nov 11 02:11:42.549: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4m32.289769913s Nov 11 02:11:44.659: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Pending", Reason="", readiness=false. Elapsed: 4m34.399784439s Nov 11 02:11:46.770: INFO: Pod "pvc-snapshottable-tester-7jn44": Phase="Succeeded", Reason="", readiness=false. Elapsed: 4m36.510388011s [1mSTEP[0m: Saw pod success Nov 11 02:11:46.770: INFO: Pod "pvc-snapshottable-tester-7jn44" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim Nov 11 02:11:46.878: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com7t57r] to have phase Bound Nov 11 02:11:46.987: INFO: PersistentVolumeClaim test.csi.azure.com7t57r found and phase=Bound (108.182636ms) [1mSTEP[0m: [init] checking the PV [1mSTEP[0m: [init] deleting the pod Nov 11 02:11:47.342: INFO: Pod pvc-snapshottable-tester-7jn44 has the following logs: ... skipping 14 lines ... Nov 11 02:11:56.991: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc creationTime:2022-11-11T02:11:52Z readyToUse:true restoreSize:5Gi] Nov 11 02:11:56.991: INFO: snapshotContentName snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc [1mSTEP[0m: checking the snapshot [1mSTEP[0m: checking the SnapshotContent [1mSTEP[0m: Modifying source data test [1mSTEP[0m: modifying the data in the source PVC Nov 11 02:11:57.428: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-4pdxb" in namespace "snapshotting-4409" to be "Succeeded or Failed" Nov 11 02:11:57.538: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 109.048383ms Nov 11 02:11:59.647: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218884206s Nov 11 02:12:01.758: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 4.32956246s Nov 11 02:12:03.867: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438686918s Nov 11 02:12:05.976: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547706872s Nov 11 02:12:08.094: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 10.665147899s ... skipping 13 lines ... Nov 11 02:12:37.632: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 40.203466877s Nov 11 02:12:39.742: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 42.313432383s Nov 11 02:12:41.852: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 44.4232079s Nov 11 02:12:43.960: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Pending", Reason="", readiness=false. Elapsed: 46.53157022s Nov 11 02:12:46.073: INFO: Pod "pvc-snapshottable-data-tester-4pdxb": Phase="Succeeded", Reason="", readiness=false. Elapsed: 48.644922411s [1mSTEP[0m: Saw pod success Nov 11 02:12:46.074: INFO: Pod "pvc-snapshottable-data-tester-4pdxb" satisfied condition "Succeeded or Failed" Nov 11 02:12:46.295: INFO: Pod pvc-snapshottable-data-tester-4pdxb has the following logs: Nov 11 02:12:46.295: INFO: Deleting pod "pvc-snapshottable-data-tester-4pdxb" in namespace "snapshotting-4409" Nov 11 02:12:46.411: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-4pdxb" to be fully deleted [1mSTEP[0m: creating a pvc from the snapshot [1mSTEP[0m: starting a pod to use the snapshot Nov 11 02:13:48.962: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-9juvoppw.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-4409 exec restored-pvc-tester-6pzsk --namespace=snapshotting-4409 -- cat /mnt/test/data' ... skipping 32 lines ... Nov 11 02:14:14.376: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted Nov 11 02:14:15.485: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted Nov 11 02:14:16.595: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted Nov 11 02:14:17.705: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted Nov 11 02:14:18.816: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted Nov 11 02:14:19.925: INFO: volumesnapshotcontents snapcontent-0cbca4ed-89d5-4398-b6a7-53888dbe0cbc has been found and is not deleted Nov 11 02:14:20.926: INFO: WaitUntil failed after reaching the timeout 30s [AfterEach] volume snapshot controller test/e2e/storage/testsuites/snapshottable.go:172 Nov 11 02:14:21.131: INFO: Pod restored-pvc-tester-6pzsk has the following logs: Nov 11 02:14:21.131: INFO: Deleting pod "restored-pvc-tester-6pzsk" in namespace "snapshotting-4409" Nov 11 02:14:21.239: INFO: Wait up to 5m0s for pod "restored-pvc-tester-6pzsk" to be fully deleted Nov 11 02:14:23.459: INFO: deleting claim "snapshotting-4409"/"pvc-49v7h" ... skipping 31 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent) [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":27,"completed":1,"skipped":76,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral test/e2e/storage/framework/testsuite.go:51 Nov 11 02:14:41.672: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping ... skipping 176 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":31,"completed":1,"skipped":148,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] provisioning test/e2e/storage/framework/testsuite.go:51 Nov 11 02:14:42.666: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 62 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.002 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m [90mtest/e2e/storage/testsuites/volumemode.go:299[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 246 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:168[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":38,"completed":1,"skipped":62,"failed":0} [36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:14:48.309: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 80 lines ... Nov 11 02:13:15.360: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comvxrcs] to have phase Bound Nov 11 02:13:15.471: INFO: PersistentVolumeClaim test.csi.azure.comvxrcs found but phase is Pending instead of Bound. Nov 11 02:13:17.578: INFO: PersistentVolumeClaim test.csi.azure.comvxrcs found but phase is Pending instead of Bound. Nov 11 02:13:19.686: INFO: PersistentVolumeClaim test.csi.azure.comvxrcs found and phase=Bound (4.325950385s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-gb7h [1mSTEP[0m: Creating a pod to test atomic-volume-subpath Nov 11 02:13:20.011: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-gb7h" in namespace "provisioning-1040" to be "Succeeded or Failed" Nov 11 02:13:20.118: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 107.238006ms Nov 11 02:13:22.230: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219190314s Nov 11 02:13:24.339: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 4.3280436s Nov 11 02:13:26.449: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 6.438010772s Nov 11 02:13:28.558: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 8.546728832s Nov 11 02:13:30.667: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655537026s ... skipping 16 lines ... Nov 11 02:14:06.532: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Running", Reason="", readiness=true. Elapsed: 46.521258636s Nov 11 02:14:08.642: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Running", Reason="", readiness=true. Elapsed: 48.631266922s Nov 11 02:14:10.756: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Running", Reason="", readiness=true. Elapsed: 50.744769319s Nov 11 02:14:12.866: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Running", Reason="", readiness=false. Elapsed: 52.855259036s Nov 11 02:14:14.977: INFO: Pod "pod-subpath-test-dynamicpv-gb7h": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.9654105s [1mSTEP[0m: Saw pod success Nov 11 02:14:14.977: INFO: Pod "pod-subpath-test-dynamicpv-gb7h" satisfied condition "Succeeded or Failed" Nov 11 02:14:15.086: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-gb7h container test-container-subpath-dynamicpv-gb7h: <nil> [1mSTEP[0m: delete the pod Nov 11 02:14:15.321: INFO: Waiting for pod pod-subpath-test-dynamicpv-gb7h to disappear Nov 11 02:14:15.429: INFO: Pod pod-subpath-test-dynamicpv-gb7h no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-gb7h Nov 11 02:14:15.429: INFO: Deleting pod "pod-subpath-test-dynamicpv-gb7h" in namespace "provisioning-1040" ... skipping 29 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support file as subpath [LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:232[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly]","total":28,"completed":2,"skipped":153,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (ext4)] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:15:27.977: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 190 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:323[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":27,"completed":2,"skipped":43,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:15:42.438: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 66 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read-only inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:175[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume","total":41,"completed":2,"skipped":260,"failed":0} [36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould support non-existent path[0m [37mtest/e2e/storage/testsuites/subpath.go:196[0m ... skipping 17 lines ... Nov 11 02:14:42.794: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comg5mrf] to have phase Bound Nov 11 02:14:42.905: INFO: PersistentVolumeClaim test.csi.azure.comg5mrf found but phase is Pending instead of Bound. Nov 11 02:14:45.014: INFO: PersistentVolumeClaim test.csi.azure.comg5mrf found but phase is Pending instead of Bound. Nov 11 02:14:47.123: INFO: PersistentVolumeClaim test.csi.azure.comg5mrf found and phase=Bound (4.328779386s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-fkj5 [1mSTEP[0m: Creating a pod to test subpath Nov 11 02:14:47.453: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-fkj5" in namespace "provisioning-561" to be "Succeeded or Failed" Nov 11 02:14:47.561: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 108.220041ms Nov 11 02:14:49.670: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217369447s Nov 11 02:14:51.779: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326264076s Nov 11 02:14:53.889: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435903032s Nov 11 02:14:55.998: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545229104s Nov 11 02:14:58.108: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655425395s ... skipping 30 lines ... Nov 11 02:16:03.514: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.060601416s Nov 11 02:16:05.632: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.17858206s Nov 11 02:16:07.751: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.298271551s Nov 11 02:16:09.861: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.408376505s Nov 11 02:16:11.971: INFO: Pod "pod-subpath-test-dynamicpv-fkj5": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m24.518055374s [1mSTEP[0m: Saw pod success Nov 11 02:16:11.971: INFO: Pod "pod-subpath-test-dynamicpv-fkj5" satisfied condition "Succeeded or Failed" Nov 11 02:16:12.078: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-fkj5 container test-container-volume-dynamicpv-fkj5: <nil> [1mSTEP[0m: delete the pod Nov 11 02:16:12.328: INFO: Waiting for pod pod-subpath-test-dynamicpv-fkj5 to disappear Nov 11 02:16:12.436: INFO: Pod pod-subpath-test-dynamicpv-fkj5 no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-fkj5 Nov 11 02:16:12.436: INFO: Deleting pod "pod-subpath-test-dynamicpv-fkj5" in namespace "provisioning-561" ... skipping 29 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support non-existent path [90mtest/e2e/storage/testsuites/subpath.go:196[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path","total":27,"completed":2,"skipped":166,"failed":0} [36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m [1mshould concurrently access the single volume from pods on the same node[0m [37mtest/e2e/storage/testsuites/multivolume.go:298[0m ... skipping 154 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:298[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":38,"completed":2,"skipped":226,"failed":0} [36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould support readOnly directory specified in the volumeMount[0m [37mtest/e2e/storage/testsuites/subpath.go:367[0m ... skipping 17 lines ... Nov 11 02:14:44.885: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6blqb] to have phase Bound Nov 11 02:14:44.995: INFO: PersistentVolumeClaim test.csi.azure.com6blqb found but phase is Pending instead of Bound. Nov 11 02:14:47.104: INFO: PersistentVolumeClaim test.csi.azure.com6blqb found but phase is Pending instead of Bound. Nov 11 02:14:49.213: INFO: PersistentVolumeClaim test.csi.azure.com6blqb found and phase=Bound (4.327944844s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-zv5m [1mSTEP[0m: Creating a pod to test subpath Nov 11 02:14:49.543: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-zv5m" in namespace "provisioning-5195" to be "Succeeded or Failed" Nov 11 02:14:49.652: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 108.742694ms Nov 11 02:14:51.763: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219215456s Nov 11 02:14:53.873: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329436267s Nov 11 02:14:55.983: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 6.439035399s Nov 11 02:14:58.093: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 8.549704534s Nov 11 02:15:00.204: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 10.660151764s ... skipping 42 lines ... Nov 11 02:16:30.965: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m41.421214198s Nov 11 02:16:33.080: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m43.536673891s Nov 11 02:16:35.189: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m45.645698135s Nov 11 02:16:37.300: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Pending", Reason="", readiness=false. Elapsed: 1m47.75611972s Nov 11 02:16:39.410: INFO: Pod "pod-subpath-test-dynamicpv-zv5m": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m49.866330048s [1mSTEP[0m: Saw pod success Nov 11 02:16:39.410: INFO: Pod "pod-subpath-test-dynamicpv-zv5m" satisfied condition "Succeeded or Failed" Nov 11 02:16:39.519: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-zv5m container test-container-subpath-dynamicpv-zv5m: <nil> [1mSTEP[0m: delete the pod Nov 11 02:16:39.746: INFO: Waiting for pod pod-subpath-test-dynamicpv-zv5m to disappear Nov 11 02:16:39.856: INFO: Pod pod-subpath-test-dynamicpv-zv5m no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-zv5m Nov 11 02:16:39.856: INFO: Deleting pod "pod-subpath-test-dynamicpv-zv5m" in namespace "provisioning-5195" ... skipping 29 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support readOnly directory specified in the volumeMount [90mtest/e2e/storage/testsuites/subpath.go:367[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount","total":31,"completed":2,"skipped":333,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumeIO test/e2e/storage/framework/testsuite.go:51 Nov 11 02:17:52.392: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 280 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext3)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should store data [90mtest/e2e/storage/testsuites/volumes.go:161[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should store data","total":38,"completed":2,"skipped":370,"failed":0} [36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow][0m [1mshould concurrently access the single volume from pods on the same node[0m [37mtest/e2e/storage/testsuites/multivolume.go:298[0m ... skipping 153 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:298[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":27,"completed":3,"skipped":163,"failed":0} [36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)(allowExpansion)][Feature:Windows] volume-expand test/e2e/storage/framework/testsuite.go:51 Nov 11 02:18:17.665: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 80 lines ... Nov 11 02:17:53.688: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.commhdpp] to have phase Bound Nov 11 02:17:53.796: INFO: PersistentVolumeClaim test.csi.azure.commhdpp found but phase is Pending instead of Bound. Nov 11 02:17:55.905: INFO: PersistentVolumeClaim test.csi.azure.commhdpp found but phase is Pending instead of Bound. Nov 11 02:17:58.013: INFO: PersistentVolumeClaim test.csi.azure.commhdpp found and phase=Bound (4.324674108s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-gmmn [1mSTEP[0m: Creating a pod to test subpath Nov 11 02:17:58.339: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-gmmn" in namespace "provisioning-3283" to be "Succeeded or Failed" Nov 11 02:17:58.447: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 107.617605ms Nov 11 02:18:00.558: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218849556s Nov 11 02:18:02.668: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328471057s Nov 11 02:18:04.776: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436761697s Nov 11 02:18:06.885: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.54552553s Nov 11 02:18:08.997: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.65771461s ... skipping 8 lines ... Nov 11 02:18:27.979: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 29.639358071s Nov 11 02:18:30.087: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 31.747857383s Nov 11 02:18:32.197: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 33.857489122s Nov 11 02:18:34.306: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Pending", Reason="", readiness=false. Elapsed: 35.966372651s Nov 11 02:18:36.414: INFO: Pod "pod-subpath-test-dynamicpv-gmmn": Phase="Succeeded", Reason="", readiness=false. Elapsed: 38.07499704s [1mSTEP[0m: Saw pod success Nov 11 02:18:36.415: INFO: Pod "pod-subpath-test-dynamicpv-gmmn" satisfied condition "Succeeded or Failed" Nov 11 02:18:36.523: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000002 pod pod-subpath-test-dynamicpv-gmmn container test-container-subpath-dynamicpv-gmmn: <nil> [1mSTEP[0m: delete the pod Nov 11 02:18:36.779: INFO: Waiting for pod pod-subpath-test-dynamicpv-gmmn to disappear Nov 11 02:18:36.886: INFO: Pod pod-subpath-test-dynamicpv-gmmn no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-gmmn Nov 11 02:18:36.886: INFO: Deleting pod "pod-subpath-test-dynamicpv-gmmn" in namespace "provisioning-3283" ... skipping 23 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support existing single file [LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:221[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly]","total":31,"completed":3,"skipped":680,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext3)] volumes[0m [1mshould allow exec of files on the volume[0m [37mtest/e2e/storage/testsuites/volumes.go:198[0m ... skipping 17 lines ... Nov 11 02:17:46.962: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comc7wqs] to have phase Bound Nov 11 02:17:47.069: INFO: PersistentVolumeClaim test.csi.azure.comc7wqs found but phase is Pending instead of Bound. Nov 11 02:17:49.177: INFO: PersistentVolumeClaim test.csi.azure.comc7wqs found but phase is Pending instead of Bound. Nov 11 02:17:51.284: INFO: PersistentVolumeClaim test.csi.azure.comc7wqs found and phase=Bound (4.322189252s) [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-xj5t [1mSTEP[0m: Creating a pod to test exec-volume-test Nov 11 02:17:51.606: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-xj5t" in namespace "volume-2914" to be "Succeeded or Failed" Nov 11 02:17:51.713: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 106.946967ms Nov 11 02:17:53.821: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 2.214463222s Nov 11 02:17:55.929: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 4.322896132s Nov 11 02:17:58.037: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 6.430980988s Nov 11 02:18:00.148: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 8.541861668s Nov 11 02:18:02.257: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 10.651195836s ... skipping 9 lines ... Nov 11 02:18:23.342: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 31.736091741s Nov 11 02:18:25.450: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 33.84435069s Nov 11 02:18:27.559: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 35.952851506s Nov 11 02:18:29.666: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Pending", Reason="", readiness=false. Elapsed: 38.060430481s Nov 11 02:18:31.776: INFO: Pod "exec-volume-test-dynamicpv-xj5t": Phase="Succeeded", Reason="", readiness=false. Elapsed: 40.169685421s [1mSTEP[0m: Saw pod success Nov 11 02:18:31.776: INFO: Pod "exec-volume-test-dynamicpv-xj5t" satisfied condition "Succeeded or Failed" Nov 11 02:18:31.883: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod exec-volume-test-dynamicpv-xj5t container exec-container-dynamicpv-xj5t: <nil> [1mSTEP[0m: delete the pod Nov 11 02:18:32.109: INFO: Waiting for pod exec-volume-test-dynamicpv-xj5t to disappear Nov 11 02:18:32.216: INFO: Pod exec-volume-test-dynamicpv-xj5t no longer exists [1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-xj5t Nov 11 02:18:32.216: INFO: Deleting pod "exec-volume-test-dynamicpv-xj5t" in namespace "volume-2914" ... skipping 27 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext3)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should allow exec of files on the volume [90mtest/e2e/storage/testsuites/volumes.go:198[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext3)] volumes should allow exec of files on the volume","total":38,"completed":3,"skipped":227,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] ... skipping 155 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should store data [90mtest/e2e/storage/testsuites/volumes.go:161[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should store data","total":28,"completed":3,"skipped":266,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral[0m [1mshould create read/write inline ephemeral volume[0m [37mtest/e2e/storage/testsuites/ephemeral.go:196[0m ... skipping 52 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read/write inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":41,"completed":3,"skipped":263,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if subpath file is outside the volume [Slow][LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:258[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Nov 11 02:18:17.759: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename provisioning [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail if subpath file is outside the volume [Slow][LinuxOnly] test/e2e/storage/testsuites/subpath.go:258 Nov 11 02:18:18.517: INFO: Creating resource for dynamic PV Nov 11 02:18:18.517: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-4691-e2e-scjrpvn [1mSTEP[0m: creating a claim Nov 11 02:18:18.626: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 11 02:18:18.737: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comkxf5f] to have phase Bound Nov 11 02:18:18.847: INFO: PersistentVolumeClaim test.csi.azure.comkxf5f found but phase is Pending instead of Bound. Nov 11 02:18:20.955: INFO: PersistentVolumeClaim test.csi.azure.comkxf5f found but phase is Pending instead of Bound. Nov 11 02:18:23.065: INFO: PersistentVolumeClaim test.csi.azure.comkxf5f found and phase=Bound (4.327903154s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-2ldb [1mSTEP[0m: Checking for subpath error in container status Nov 11 02:19:05.608: INFO: Deleting pod "pod-subpath-test-dynamicpv-2ldb" in namespace "provisioning-4691" Nov 11 02:19:05.717: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-2ldb" to be fully deleted [1mSTEP[0m: Deleting pod Nov 11 02:19:07.935: INFO: Deleting pod "pod-subpath-test-dynamicpv-2ldb" in namespace "provisioning-4691" [1mSTEP[0m: Deleting pvc Nov 11 02:19:08.055: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comkxf5f" ... skipping 21 lines ... [32m• [SLOW TEST:117.482 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail if subpath file is outside the volume [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:258[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow][LinuxOnly]","total":27,"completed":4,"skipped":254,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould support readOnly file specified in the volumeMount [LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:382[0m ... skipping 17 lines ... Nov 11 02:18:04.946: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comq6h28] to have phase Bound Nov 11 02:18:05.053: INFO: PersistentVolumeClaim test.csi.azure.comq6h28 found but phase is Pending instead of Bound. Nov 11 02:18:07.162: INFO: PersistentVolumeClaim test.csi.azure.comq6h28 found but phase is Pending instead of Bound. Nov 11 02:18:09.271: INFO: PersistentVolumeClaim test.csi.azure.comq6h28 found and phase=Bound (4.325051455s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-5m7s [1mSTEP[0m: Creating a pod to test subpath Nov 11 02:18:09.597: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-5m7s" in namespace "provisioning-9412" to be "Succeeded or Failed" Nov 11 02:18:09.707: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 109.734964ms Nov 11 02:18:11.816: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219040665s Nov 11 02:18:13.926: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328360631s Nov 11 02:18:16.035: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437920156s Nov 11 02:18:18.144: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 8.5465115s Nov 11 02:18:20.254: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657182535s ... skipping 18 lines ... Nov 11 02:19:00.330: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 50.732786849s Nov 11 02:19:02.440: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 52.842873182s Nov 11 02:19:04.549: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 54.951823676s Nov 11 02:19:06.659: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Pending", Reason="", readiness=false. Elapsed: 57.062034366s Nov 11 02:19:08.768: INFO: Pod "pod-subpath-test-dynamicpv-5m7s": Phase="Succeeded", Reason="", readiness=false. Elapsed: 59.170776864s [1mSTEP[0m: Saw pod success Nov 11 02:19:08.768: INFO: Pod "pod-subpath-test-dynamicpv-5m7s" satisfied condition "Succeeded or Failed" Nov 11 02:19:08.876: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod pod-subpath-test-dynamicpv-5m7s container test-container-subpath-dynamicpv-5m7s: <nil> [1mSTEP[0m: delete the pod Nov 11 02:19:09.129: INFO: Waiting for pod pod-subpath-test-dynamicpv-5m7s to disappear Nov 11 02:19:09.236: INFO: Pod pod-subpath-test-dynamicpv-5m7s no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-5m7s Nov 11 02:19:09.236: INFO: Deleting pod "pod-subpath-test-dynamicpv-5m7s" in namespace "provisioning-9412" ... skipping 29 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support readOnly file specified in the volumeMount [LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:382[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly]","total":38,"completed":3,"skipped":373,"failed":0} [36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] volumeIO[0m [1mshould write files of various sizes, verify size, validate content [Slow][0m [37mtest/e2e/storage/testsuites/volume_io.go:149[0m ... skipping 48 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] volumeIO [90mtest/e2e/storage/framework/testsuite.go:50[0m should write files of various sizes, verify size, validate content [Slow] [90mtest/e2e/storage/testsuites/volume_io.go:149[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow]","total":31,"completed":4,"skipped":737,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:20:53.039: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 171 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should store data [90mtest/e2e/storage/testsuites/volumes.go:161[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] volumes should store data","total":27,"completed":3,"skipped":169,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould support existing directories when readOnly specified in the volumeSource[0m [37mtest/e2e/storage/testsuites/subpath.go:397[0m ... skipping 17 lines ... Nov 11 02:19:51.060: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comlpk4k] to have phase Bound Nov 11 02:19:51.167: INFO: PersistentVolumeClaim test.csi.azure.comlpk4k found but phase is Pending instead of Bound. Nov 11 02:19:53.275: INFO: PersistentVolumeClaim test.csi.azure.comlpk4k found but phase is Pending instead of Bound. Nov 11 02:19:55.382: INFO: PersistentVolumeClaim test.csi.azure.comlpk4k found and phase=Bound (4.321890674s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-flfm [1mSTEP[0m: Creating a pod to test subpath Nov 11 02:19:55.707: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-flfm" in namespace "provisioning-9676" to be "Succeeded or Failed" Nov 11 02:19:55.815: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 107.309531ms Nov 11 02:19:57.922: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.214677108s Nov 11 02:20:00.030: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.323015759s Nov 11 02:20:02.139: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43195492s Nov 11 02:20:04.247: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.539684753s Nov 11 02:20:06.355: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.647568499s Nov 11 02:20:08.463: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 12.75525108s Nov 11 02:20:10.570: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 14.86273569s Nov 11 02:20:12.679: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 16.971423048s Nov 11 02:20:14.786: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 19.078377928s Nov 11 02:20:16.893: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 21.185304565s Nov 11 02:20:19.002: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 23.294482383s [1mSTEP[0m: Saw pod success Nov 11 02:20:19.002: INFO: Pod "pod-subpath-test-dynamicpv-flfm" satisfied condition "Succeeded or Failed" Nov 11 02:20:19.109: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod pod-subpath-test-dynamicpv-flfm container test-container-subpath-dynamicpv-flfm: <nil> [1mSTEP[0m: delete the pod Nov 11 02:20:19.337: INFO: Waiting for pod pod-subpath-test-dynamicpv-flfm to disappear Nov 11 02:20:19.444: INFO: Pod pod-subpath-test-dynamicpv-flfm no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-flfm Nov 11 02:20:19.444: INFO: Deleting pod "pod-subpath-test-dynamicpv-flfm" in namespace "provisioning-9676" [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-flfm [1mSTEP[0m: Creating a pod to test subpath Nov 11 02:20:19.659: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-flfm" in namespace "provisioning-9676" to be "Succeeded or Failed" Nov 11 02:20:19.766: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 106.759893ms Nov 11 02:20:21.881: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 2.221593371s Nov 11 02:20:23.989: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 4.329607167s Nov 11 02:20:26.097: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 6.437639598s Nov 11 02:20:28.205: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545499816s Nov 11 02:20:30.313: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653403397s ... skipping 11 lines ... Nov 11 02:20:55.622: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 35.962436064s Nov 11 02:20:57.730: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 38.070574012s Nov 11 02:20:59.839: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 40.179852682s Nov 11 02:21:01.947: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Pending", Reason="", readiness=false. Elapsed: 42.287459786s Nov 11 02:21:04.055: INFO: Pod "pod-subpath-test-dynamicpv-flfm": Phase="Succeeded", Reason="", readiness=false. Elapsed: 44.396166768s [1mSTEP[0m: Saw pod success Nov 11 02:21:04.056: INFO: Pod "pod-subpath-test-dynamicpv-flfm" satisfied condition "Succeeded or Failed" Nov 11 02:21:04.163: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod pod-subpath-test-dynamicpv-flfm container test-container-subpath-dynamicpv-flfm: <nil> [1mSTEP[0m: delete the pod Nov 11 02:21:04.390: INFO: Waiting for pod pod-subpath-test-dynamicpv-flfm to disappear Nov 11 02:21:04.497: INFO: Pod pod-subpath-test-dynamicpv-flfm no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-flfm Nov 11 02:21:04.497: INFO: Deleting pod "pod-subpath-test-dynamicpv-flfm" in namespace "provisioning-9676" ... skipping 26 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support existing directories when readOnly specified in the volumeSource [90mtest/e2e/storage/testsuites/subpath.go:397[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource","total":28,"completed":4,"skipped":321,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m [1mshould concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS][0m [37mtest/e2e/storage/testsuites/multivolume.go:378[0m ... skipping 79 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:378[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":41,"completed":4,"skipped":284,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:22:29.136: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:280[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 221 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:209[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":27,"completed":5,"skipped":269,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral test/e2e/storage/framework/testsuite.go:51 Nov 11 02:23:36.893: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping ... skipping 197 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:378[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":27,"completed":4,"skipped":228,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode test/e2e/storage/framework/testsuite.go:51 Nov 11 02:24:52.139: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 24 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Inline-volume (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:242[0m [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 145 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] [90mtest/e2e/storage/testsuites/provisioning.go:208[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":38,"completed":4,"skipped":374,"failed":0} [36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral[0m [1mshould support two pods which have the same volume definition[0m [37mtest/e2e/storage/testsuites/ephemeral.go:216[0m ... skipping 63 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support two pods which have the same volume definition [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition","total":28,"completed":5,"skipped":350,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:25:15.530: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 216 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:209[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":41,"completed":5,"skipped":355,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow][0m [1mshould access to two volumes with different volume mode and retain data across pod recreation on different node[0m [37mtest/e2e/storage/testsuites/multivolume.go:248[0m ... skipping 195 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:248[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":31,"completed":5,"skipped":833,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould support existing directory[0m [37mtest/e2e/storage/testsuites/subpath.go:207[0m ... skipping 17 lines ... Nov 11 02:23:37.862: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comqd76d] to have phase Bound Nov 11 02:23:37.969: INFO: PersistentVolumeClaim test.csi.azure.comqd76d found but phase is Pending instead of Bound. Nov 11 02:23:40.077: INFO: PersistentVolumeClaim test.csi.azure.comqd76d found but phase is Pending instead of Bound. Nov 11 02:23:42.184: INFO: PersistentVolumeClaim test.csi.azure.comqd76d found and phase=Bound (4.321599989s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-292d [1mSTEP[0m: Creating a pod to test subpath Nov 11 02:23:42.507: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-292d" in namespace "provisioning-9310" to be "Succeeded or Failed" Nov 11 02:23:42.615: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 107.036851ms Nov 11 02:23:44.723: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 2.215453988s Nov 11 02:23:46.830: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 4.322997243s Nov 11 02:23:48.938: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 6.430521575s Nov 11 02:23:51.047: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 8.539623983s Nov 11 02:23:53.154: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 10.646753219s ... skipping 6 lines ... Nov 11 02:24:07.912: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 25.404866305s Nov 11 02:24:10.021: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 27.51346437s Nov 11 02:24:12.129: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 29.621556019s Nov 11 02:24:14.238: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Pending", Reason="", readiness=false. Elapsed: 31.730095719s Nov 11 02:24:16.345: INFO: Pod "pod-subpath-test-dynamicpv-292d": Phase="Succeeded", Reason="", readiness=false. Elapsed: 33.837864371s [1mSTEP[0m: Saw pod success Nov 11 02:24:16.345: INFO: Pod "pod-subpath-test-dynamicpv-292d" satisfied condition "Succeeded or Failed" Nov 11 02:24:16.453: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-292d container test-container-volume-dynamicpv-292d: <nil> [1mSTEP[0m: delete the pod Nov 11 02:24:16.711: INFO: Waiting for pod pod-subpath-test-dynamicpv-292d to disappear Nov 11 02:24:16.818: INFO: Pod pod-subpath-test-dynamicpv-292d no longer exists [1mSTEP[0m: Deleting pod pod-subpath-test-dynamicpv-292d Nov 11 02:24:16.818: INFO: Deleting pod "pod-subpath-test-dynamicpv-292d" in namespace "provisioning-9310" ... skipping 29 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support existing directory [90mtest/e2e/storage/testsuites/subpath.go:207[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory","total":27,"completed":6,"skipped":284,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:25:29.371: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 157 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support restarting containers using file as subpath [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:333[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using file as subpath [Slow][LinuxOnly]","total":27,"completed":5,"skipped":397,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:29:05.287: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 193 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:298[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":27,"completed":7,"skipped":506,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits[0m [1mshould verify that all csinodes have volume limits[0m [37mtest/e2e/storage/testsuites/volumelimits.go:249[0m ... skipping 16 lines ... test/e2e/framework/framework.go:188 Nov 11 02:29:32.948: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [1mSTEP[0m: Destroying namespace "volumelimits-159" for this suite. [32m•[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs)] volumeLimits should verify that all csinodes have volume limits","total":27,"completed":8,"skipped":578,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:29:33.190: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 13 lines ... test/e2e/storage/external/external.go:262 [90m------------------------------[0m [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:242[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":38,"completed":4,"skipped":282,"failed":0} [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Nov 11 02:23:56.680: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename provisioning [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail if subpath directory is outside the volume [Slow][LinuxOnly] test/e2e/storage/testsuites/subpath.go:242 Nov 11 02:23:57.432: INFO: Creating resource for dynamic PV Nov 11 02:23:57.432: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-2722-e2e-scjxc8x [1mSTEP[0m: creating a claim Nov 11 02:23:57.540: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 11 02:23:57.651: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comjw8q9] to have phase Bound Nov 11 02:23:57.758: INFO: PersistentVolumeClaim test.csi.azure.comjw8q9 found but phase is Pending instead of Bound. Nov 11 02:23:59.867: INFO: PersistentVolumeClaim test.csi.azure.comjw8q9 found but phase is Pending instead of Bound. Nov 11 02:24:01.975: INFO: PersistentVolumeClaim test.csi.azure.comjw8q9 found and phase=Bound (4.324576077s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-qqgq [1mSTEP[0m: Checking for subpath error in container status Nov 11 02:25:14.515: INFO: Deleting pod "pod-subpath-test-dynamicpv-qqgq" in namespace "provisioning-2722" Nov 11 02:25:14.624: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-qqgq" to be fully deleted [1mSTEP[0m: Deleting pod Nov 11 02:25:16.839: INFO: Deleting pod "pod-subpath-test-dynamicpv-qqgq" in namespace "provisioning-2722" [1mSTEP[0m: Deleting pvc Nov 11 02:25:16.947: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comjw8q9" ... skipping 59 lines ... [32m• [SLOW TEST:341.605 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail if subpath directory is outside the volume [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:242[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow][LinuxOnly]","total":38,"completed":5,"skipped":282,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral[0m [1mshould support multiple inline ephemeral volumes[0m [37mtest/e2e/storage/testsuites/ephemeral.go:254[0m ... skipping 51 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support multiple inline ephemeral volumes [90mtest/e2e/storage/testsuites/ephemeral.go:254[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":28,"completed":6,"skipped":394,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (ext3)] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:30:26.696: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 245 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:248[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":31,"completed":6,"skipped":850,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral[0m [1mshould create read-only inline ephemeral volume[0m [37mtest/e2e/storage/testsuites/ephemeral.go:175[0m ... skipping 52 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read-only inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:175[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume","total":27,"completed":9,"skipped":610,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Nov 11 02:31:49.004: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename topology [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies test/e2e/storage/testsuites/topology.go:194 Nov 11 02:31:49.759: INFO: Driver didn't provide topology keys -- skipping [AfterEach] [Testpattern: Dynamic PV (immediate binding)] topology test/e2e/framework/framework.go:188 Nov 11 02:31:49.759: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [1mSTEP[0m: Destroying namespace "topology-6842" for this suite. [36m[1mS [SKIPPING] [1.080 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (immediate binding)] topology [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement][0m [90mtest/e2e/storage/testsuites/topology.go:194[0m [36mDriver didn't provide topology keys -- skipping[0m test/e2e/storage/testsuites/topology.go:126 [90m------------------------------[0m ... skipping 104 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single read-only volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:423[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":27,"completed":6,"skipped":508,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Nov 11 02:32:32.163: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 227 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:168[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":41,"completed":6,"skipped":376,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m [1mshould access to two volumes with the same volume mode and retain data across pod recreation on the same node[0m [37mtest/e2e/storage/testsuites/multivolume.go:138[0m ... skipping 206 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:138[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":38,"completed":5,"skipped":375,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:33:33.292: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 192 lines ... Nov 11 02:31:51.116: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com6n7j4] to have phase Bound Nov 11 02:31:51.227: INFO: PersistentVolumeClaim test.csi.azure.com6n7j4 found but phase is Pending instead of Bound. Nov 11 02:31:53.337: INFO: PersistentVolumeClaim test.csi.azure.com6n7j4 found but phase is Pending instead of Bound. Nov 11 02:31:55.446: INFO: PersistentVolumeClaim test.csi.azure.com6n7j4 found and phase=Bound (4.329889029s) [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-sl65 [1mSTEP[0m: Creating a pod to test exec-volume-test Nov 11 02:31:55.774: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-sl65" in namespace "volume-7259" to be "Succeeded or Failed" Nov 11 02:31:55.884: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 110.0421ms Nov 11 02:31:57.993: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 2.218667676s Nov 11 02:32:00.102: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327999878s Nov 11 02:32:02.212: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43755528s Nov 11 02:32:04.322: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 8.54759254s Nov 11 02:32:06.432: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 10.657437246s ... skipping 29 lines ... Nov 11 02:33:09.718: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 1m13.943878314s Nov 11 02:33:11.829: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 1m16.054575768s Nov 11 02:33:13.940: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 1m18.16533005s Nov 11 02:33:16.048: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Pending", Reason="", readiness=false. Elapsed: 1m20.274031565s Nov 11 02:33:18.157: INFO: Pod "exec-volume-test-dynamicpv-sl65": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m22.382642458s [1mSTEP[0m: Saw pod success Nov 11 02:33:18.157: INFO: Pod "exec-volume-test-dynamicpv-sl65" satisfied condition "Succeeded or Failed" Nov 11 02:33:18.265: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod exec-volume-test-dynamicpv-sl65 container exec-container-dynamicpv-sl65: <nil> [1mSTEP[0m: delete the pod Nov 11 02:33:18.514: INFO: Waiting for pod exec-volume-test-dynamicpv-sl65 to disappear Nov 11 02:33:18.623: INFO: Pod exec-volume-test-dynamicpv-sl65 no longer exists [1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-sl65 Nov 11 02:33:18.623: INFO: Deleting pod "exec-volume-test-dynamicpv-sl65" in namespace "volume-7259" ... skipping 27 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should allow exec of files on the volume [90mtest/e2e/storage/testsuites/volumes.go:198[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should allow exec of files on the volume","total":27,"completed":10,"skipped":704,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (xfs)][Slow] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:34:31.034: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 253 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m should not mount / map unused volumes in a pod [LinuxOnly] [90mtest/e2e/storage/testsuites/volumemode.go:354[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]","total":31,"completed":7,"skipped":908,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] volumeLimits[0m [1mshould verify that all csinodes have volume limits[0m [37mtest/e2e/storage/testsuites/volumelimits.go:249[0m ... skipping 16 lines ... test/e2e/framework/framework.go:188 Nov 11 02:35:35.688: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [1mSTEP[0m: Destroying namespace "volumelimits-2896" for this suite. [32m•[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeLimits should verify that all csinodes have volume limits","total":31,"completed":8,"skipped":942,"failed":1,"failures":["External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly]"]} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Nov 11 02:35:35.925: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 175 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should store data [90mtest/e2e/storage/testsuites/volumes.go:161[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumes should store data","total":38,"completed":6,"skipped":646,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (ext3)] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:35:52.804: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 363 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with pvc data source in parallel [Slow] [90mtest/e2e/storage/testsuites/provisioning.go:459[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source in parallel [Slow]","total":27,"completed":7,"skipped":576,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:36:00.934: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 141 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:323[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":38,"completed":6,"skipped":310,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:280[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Nov 11 02:35:52.864: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename provisioning [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] test/e2e/storage/testsuites/subpath.go:280 Nov 11 02:35:53.620: INFO: Creating resource for dynamic PV Nov 11 02:35:53.620: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-127-e2e-sccgj64 [1mSTEP[0m: creating a claim Nov 11 02:35:53.728: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 11 02:35:53.839: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comkk9cv] to have phase Bound Nov 11 02:35:53.947: INFO: PersistentVolumeClaim test.csi.azure.comkk9cv found but phase is Pending instead of Bound. Nov 11 02:35:56.056: INFO: PersistentVolumeClaim test.csi.azure.comkk9cv found but phase is Pending instead of Bound. Nov 11 02:35:58.165: INFO: PersistentVolumeClaim test.csi.azure.comkk9cv found and phase=Bound (4.326052114s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-t2s6 [1mSTEP[0m: Checking for subpath error in container status Nov 11 02:36:16.711: INFO: Deleting pod "pod-subpath-test-dynamicpv-t2s6" in namespace "provisioning-127" Nov 11 02:36:16.822: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-t2s6" to be fully deleted [1mSTEP[0m: Deleting pod Nov 11 02:36:19.039: INFO: Deleting pod "pod-subpath-test-dynamicpv-t2s6" in namespace "provisioning-127" [1mSTEP[0m: Deleting pvc Nov 11 02:36:19.147: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comkk9cv" ... skipping 12 lines ... [32m• [SLOW TEST:47.477 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:280[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly]","total":38,"completed":7,"skipped":712,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource][0m [0mvolume snapshot controller[0m [90m[0m [1mshould check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)[0m [37mtest/e2e/storage/testsuites/snapshottable.go:278[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":28,"completed":7,"skipped":614,"failed":0} [BeforeEach] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Nov 11 02:34:14.628: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json ... skipping 10 lines ... Nov 11 02:34:15.599: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comwmppf] to have phase Bound Nov 11 02:34:15.707: INFO: PersistentVolumeClaim test.csi.azure.comwmppf found but phase is Pending instead of Bound. Nov 11 02:34:17.816: INFO: PersistentVolumeClaim test.csi.azure.comwmppf found but phase is Pending instead of Bound. Nov 11 02:34:19.924: INFO: PersistentVolumeClaim test.csi.azure.comwmppf found and phase=Bound (4.324920315s) [1mSTEP[0m: [init] starting a pod to use the claim [1mSTEP[0m: [init] check pod success Nov 11 02:34:20.359: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-ng9td" in namespace "snapshotting-4341" to be "Succeeded or Failed" Nov 11 02:34:20.467: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 108.003837ms Nov 11 02:34:22.576: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217358408s Nov 11 02:34:24.685: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325770563s Nov 11 02:34:26.794: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43501537s Nov 11 02:34:28.905: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545881275s Nov 11 02:34:31.015: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 10.655694946s Nov 11 02:34:33.123: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 12.763886413s Nov 11 02:34:35.230: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Pending", Reason="", readiness=false. Elapsed: 14.871507913s Nov 11 02:34:37.347: INFO: Pod "pvc-snapshottable-tester-ng9td": Phase="Succeeded", Reason="", readiness=false. Elapsed: 16.987807798s [1mSTEP[0m: Saw pod success Nov 11 02:34:37.347: INFO: Pod "pvc-snapshottable-tester-ng9td" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim Nov 11 02:34:37.455: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comwmppf] to have phase Bound Nov 11 02:34:37.562: INFO: PersistentVolumeClaim test.csi.azure.comwmppf found and phase=Bound (107.403429ms) [1mSTEP[0m: [init] checking the PV [1mSTEP[0m: [init] deleting the pod Nov 11 02:34:37.912: INFO: Pod pvc-snapshottable-tester-ng9td has the following logs: ... skipping 14 lines ... Nov 11 02:34:47.552: INFO: received snapshotStatus map[boundVolumeSnapshotContentName:snapcontent-9cfb2f8c-b3c6-4d25-bfbe-0d82e90a57a2 creationTime:2022-11-11T02:34:43Z readyToUse:true restoreSize:5Gi] Nov 11 02:34:47.552: INFO: snapshotContentName snapcontent-9cfb2f8c-b3c6-4d25-bfbe-0d82e90a57a2 [1mSTEP[0m: checking the snapshot [1mSTEP[0m: checking the SnapshotContent [1mSTEP[0m: Modifying source data test [1mSTEP[0m: modifying the data in the source PVC Nov 11 02:34:47.998: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-5wjng" in namespace "snapshotting-4341" to be "Succeeded or Failed" Nov 11 02:34:48.106: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 107.821484ms Nov 11 02:34:50.215: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216907877s Nov 11 02:34:52.324: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326360323s Nov 11 02:34:54.433: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435187992s Nov 11 02:34:56.542: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 8.543664525s Nov 11 02:34:58.650: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 10.652380336s ... skipping 21 lines ... Nov 11 02:35:45.067: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 57.06892718s Nov 11 02:35:47.175: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 59.177239895s Nov 11 02:35:49.283: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.285407094s Nov 11 02:35:51.391: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.393288198s Nov 11 02:35:53.500: INFO: Pod "pvc-snapshottable-data-tester-5wjng": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m5.501541883s [1mSTEP[0m: Saw pod success Nov 11 02:35:53.500: INFO: Pod "pvc-snapshottable-data-tester-5wjng" satisfied condition "Succeeded or Failed" Nov 11 02:35:53.749: INFO: Pod pvc-snapshottable-data-tester-5wjng has the following logs: Nov 11 02:35:53.749: INFO: Deleting pod "pvc-snapshottable-data-tester-5wjng" in namespace "snapshotting-4341" Nov 11 02:35:53.867: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-5wjng" to be fully deleted [1mSTEP[0m: creating a pvc from the snapshot [1mSTEP[0m: starting a pod to use the snapshot Nov 11 02:36:18.415: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-9juvoppw.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-4341 exec restored-pvc-tester-frtr8 --namespace=snapshotting-4341 -- cat /mnt/test/data' ... skipping 47 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent) [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":28,"completed":8,"skipped":614,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] ... skipping 226 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:248[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node","total":41,"completed":7,"skipped":396,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:37:14.929: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 187 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents","total":27,"completed":8,"skipped":648,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Nov 11 02:37:54.708: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 125 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:242[0m [36mDistro debian doesn't support ntfs -- skipping[0m test/e2e/storage/framework/testsuite.go:127 [90m------------------------------[0m ... skipping 96 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":38,"completed":8,"skipped":719,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volume-expand test/e2e/storage/framework/testsuite.go:51 Nov 11 02:38:18.646: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 155 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents","total":38,"completed":7,"skipped":314,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:38:55.060: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 13 lines ... test/e2e/storage/external/external.go:262 [90m------------------------------[0m [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:269[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Nov 11 02:37:15.993: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename provisioning [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] test/e2e/storage/testsuites/subpath.go:269 Nov 11 02:37:16.740: INFO: Creating resource for dynamic PV Nov 11 02:37:16.740: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-9449-e2e-sc55khs [1mSTEP[0m: creating a claim Nov 11 02:37:16.848: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 11 02:37:16.957: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comjf52l] to have phase Bound Nov 11 02:37:17.063: INFO: PersistentVolumeClaim test.csi.azure.comjf52l found but phase is Pending instead of Bound. Nov 11 02:37:19.172: INFO: PersistentVolumeClaim test.csi.azure.comjf52l found but phase is Pending instead of Bound. Nov 11 02:37:21.280: INFO: PersistentVolumeClaim test.csi.azure.comjf52l found and phase=Bound (4.322959341s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-85l8 [1mSTEP[0m: Checking for subpath error in container status Nov 11 02:37:51.819: INFO: Deleting pod "pod-subpath-test-dynamicpv-85l8" in namespace "provisioning-9449" Nov 11 02:37:51.927: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-85l8" to be fully deleted [1mSTEP[0m: Deleting pod Nov 11 02:37:54.142: INFO: Deleting pod "pod-subpath-test-dynamicpv-85l8" in namespace "provisioning-9449" [1mSTEP[0m: Deleting pvc Nov 11 02:37:54.250: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comjf52l" ... skipping 22 lines ... [32m• [SLOW TEST:110.566 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:269[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]","total":41,"completed":8,"skipped":519,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Nov 11 02:39:06.565: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename topology [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies test/e2e/storage/testsuites/topology.go:194 Nov 11 02:39:07.319: INFO: Driver didn't provide topology keys -- skipping [AfterEach] [Testpattern: Dynamic PV (delayed binding)] topology test/e2e/framework/framework.go:188 Nov 11 02:39:07.319: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready [1mSTEP[0m: Destroying namespace "topology-9817" for this suite. [36m[1mS [SKIPPING] [1.077 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (delayed binding)] topology [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies [Measurement][0m [90mtest/e2e/storage/testsuites/topology.go:194[0m [36mDriver didn't provide topology keys -- skipping[0m test/e2e/storage/testsuites/topology.go:126 [90m------------------------------[0m ... skipping 216 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:138[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on the same node","total":28,"completed":9,"skipped":651,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould be able to unmount after the subpath directory is deleted [LinuxOnly][0m [37mtest/e2e/storage/testsuites/subpath.go:447[0m ... skipping 57 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should be able to unmount after the subpath directory is deleted [LinuxOnly] [90mtest/e2e/storage/testsuites/subpath.go:447[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly]","total":38,"completed":9,"skipped":767,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] ... skipping 164 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single read-only volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:423[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":27,"completed":9,"skipped":833,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:40:31.306: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 50 lines ... [It] should check snapshot fields, check restore correctly works, check deletion (ephemeral) test/e2e/storage/testsuites/snapshottable.go:177 Nov 11 02:38:55.905: INFO: Creating resource for dynamic PV Nov 11 02:38:55.905: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass snapshotting-3890-e2e-sc8skz4 [1mSTEP[0m: [init] starting a pod to use the claim Nov 11 02:38:56.132: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-dcj5r" in namespace "snapshotting-3890" to be "Succeeded or Failed" Nov 11 02:38:56.240: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 108.052397ms Nov 11 02:38:58.349: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217528287s Nov 11 02:39:00.459: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327553332s Nov 11 02:39:02.568: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 6.436440015s Nov 11 02:39:04.677: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545542001s Nov 11 02:39:06.786: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653956435s ... skipping 3 lines ... Nov 11 02:39:15.225: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 19.093501975s Nov 11 02:39:17.335: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 21.203283437s Nov 11 02:39:19.445: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 23.312934475s Nov 11 02:39:21.554: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Pending", Reason="", readiness=false. Elapsed: 25.421835726s Nov 11 02:39:23.662: INFO: Pod "pvc-snapshottable-tester-dcj5r": Phase="Succeeded", Reason="", readiness=false. Elapsed: 27.530721068s [1mSTEP[0m: Saw pod success Nov 11 02:39:23.662: INFO: Pod "pvc-snapshottable-tester-dcj5r" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim [1mSTEP[0m: creating a SnapshotClass [1mSTEP[0m: creating a dynamic VolumeSnapshot Nov 11 02:39:24.101: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-9hcb6 to become ready Nov 11 02:39:24.209: INFO: VolumeSnapshot snapshot-9hcb6 found but is not ready. Nov 11 02:39:26.319: INFO: VolumeSnapshot snapshot-9hcb6 found but is not ready. ... skipping 49 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works, check deletion (ephemeral) [90mtest/e2e/storage/testsuites/snapshottable.go:177[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":38,"completed":8,"skipped":445,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:40:53.854: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 24 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:242[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 156 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support restarting containers using directory as subpath [Slow] [90mtest/e2e/storage/testsuites/subpath.go:322[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support restarting containers using directory as subpath [Slow]","total":41,"completed":9,"skipped":637,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:42:47.505: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 73 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read/write inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume","total":38,"completed":9,"skipped":578,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource][0m [0mvolume snapshot controller[0m [90m[0m [1mshould check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)[0m [37mtest/e2e/storage/testsuites/snapshottable.go:278[0m ... skipping 17 lines ... Nov 11 02:39:15.258: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combwbhs] to have phase Bound Nov 11 02:39:15.366: INFO: PersistentVolumeClaim test.csi.azure.combwbhs found but phase is Pending instead of Bound. Nov 11 02:39:17.474: INFO: PersistentVolumeClaim test.csi.azure.combwbhs found but phase is Pending instead of Bound. Nov 11 02:39:19.583: INFO: PersistentVolumeClaim test.csi.azure.combwbhs found and phase=Bound (4.324754524s) [1mSTEP[0m: [init] starting a pod to use the claim [1mSTEP[0m: [init] check pod success Nov 11 02:39:20.016: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-z7gpt" in namespace "snapshotting-955" to be "Succeeded or Failed" Nov 11 02:39:20.124: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 107.828133ms Nov 11 02:39:22.233: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21743853s Nov 11 02:39:24.343: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 4.327124906s Nov 11 02:39:26.458: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 6.441766709s Nov 11 02:39:28.567: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 8.551486557s Nov 11 02:39:30.676: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 10.659882193s ... skipping 22 lines ... Nov 11 02:40:19.215: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 59.199081097s Nov 11 02:40:21.325: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m1.309019171s Nov 11 02:40:23.433: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m3.417167317s Nov 11 02:40:25.543: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.526579133s Nov 11 02:40:27.652: INFO: Pod "pvc-snapshottable-tester-z7gpt": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m7.635742965s [1mSTEP[0m: Saw pod success Nov 11 02:40:27.652: INFO: Pod "pvc-snapshottable-tester-z7gpt" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim Nov 11 02:40:27.760: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combwbhs] to have phase Bound Nov 11 02:40:27.869: INFO: PersistentVolumeClaim test.csi.azure.combwbhs found and phase=Bound (108.928842ms) [1mSTEP[0m: [init] checking the PV [1mSTEP[0m: [init] deleting the pod Nov 11 02:40:28.221: INFO: Pod pvc-snapshottable-tester-z7gpt has the following logs: ... skipping 37 lines ... Nov 11 02:40:45.485: INFO: WaitUntil finished successfully after 108.729202ms [1mSTEP[0m: getting the snapshot and snapshot content [1mSTEP[0m: checking the snapshot [1mSTEP[0m: checking the SnapshotContent [1mSTEP[0m: Modifying source data test [1mSTEP[0m: modifying the data in the source PVC Nov 11 02:40:46.029: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-zbnw9" in namespace "snapshotting-955" to be "Succeeded or Failed" Nov 11 02:40:46.136: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 107.702518ms Nov 11 02:40:48.246: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217101798s Nov 11 02:40:50.354: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325598452s Nov 11 02:40:52.462: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.433478998s Nov 11 02:40:54.571: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.542553515s Nov 11 02:40:56.681: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.652204926s ... skipping 33 lines ... Nov 11 02:42:08.435: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 1m22.406259253s Nov 11 02:42:10.544: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 1m24.514953457s Nov 11 02:42:12.654: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 1m26.624900306s Nov 11 02:42:14.763: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Pending", Reason="", readiness=false. Elapsed: 1m28.733750842s Nov 11 02:42:16.871: INFO: Pod "pvc-snapshottable-data-tester-zbnw9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m30.842305376s [1mSTEP[0m: Saw pod success Nov 11 02:42:16.871: INFO: Pod "pvc-snapshottable-data-tester-zbnw9" satisfied condition "Succeeded or Failed" Nov 11 02:42:17.091: INFO: Pod pvc-snapshottable-data-tester-zbnw9 has the following logs: Nov 11 02:42:17.091: INFO: Deleting pod "pvc-snapshottable-data-tester-zbnw9" in namespace "snapshotting-955" Nov 11 02:42:17.201: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-zbnw9" to be fully deleted [1mSTEP[0m: creating a pvc from the snapshot [1mSTEP[0m: starting a pod to use the snapshot Nov 11 02:43:43.749: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-9juvoppw.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-955 exec restored-pvc-tester-g7lx8 --namespace=snapshotting-955 -- cat /mnt/test/data' ... skipping 48 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent) [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":28,"completed":10,"skipped":716,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] capacity test/e2e/storage/framework/testsuite.go:51 Nov 11 02:44:33.169: INFO: Driver test.csi.azure.com doesn't publish storage capacity -- skipping ... skipping 142 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:323[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should concurrently access the volume and restored snapshot from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":38,"completed":10,"skipped":789,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] ... skipping 30 lines ... test/e2e/storage/testsuites/snapshottable.go:179 [90m------------------------------[0m [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] volumeMode[0m [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m [37mtest/e2e/storage/testsuites/volumemode.go:299[0m [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Nov 11 02:42:47.535: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename volumemode [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail to use a volume in a pod with mismatched mode [Slow] test/e2e/storage/testsuites/volumemode.go:299 Nov 11 02:42:48.288: INFO: Creating resource for dynamic PV Nov 11 02:42:48.288: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass volumemode-2382-e2e-sc24hx2 [1mSTEP[0m: creating a claim Nov 11 02:42:48.507: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comlc6tg] to have phase Bound Nov 11 02:42:48.615: INFO: PersistentVolumeClaim test.csi.azure.comlc6tg found but phase is Pending instead of Bound. Nov 11 02:42:50.724: INFO: PersistentVolumeClaim test.csi.azure.comlc6tg found but phase is Pending instead of Bound. Nov 11 02:42:52.834: INFO: PersistentVolumeClaim test.csi.azure.comlc6tg found and phase=Bound (4.32651136s) [1mSTEP[0m: Creating pod [1mSTEP[0m: Waiting for the pod to fail Nov 11 02:42:55.484: INFO: Deleting pod "pod-a6809ccf-5ae9-4cef-a7ad-18cad857459b" in namespace "volumemode-2382" Nov 11 02:42:55.593: INFO: Wait up to 5m0s for pod "pod-a6809ccf-5ae9-4cef-a7ad-18cad857459b" to be fully deleted [1mSTEP[0m: Deleting pvc Nov 11 02:42:57.809: INFO: Deleting PersistentVolumeClaim "test.csi.azure.comlc6tg" Nov 11 02:42:57.918: INFO: Waiting up to 5m0s for PersistentVolume pvc-1fb3e23a-88f4-4267-af34-aeba0686f77f to get deleted Nov 11 02:42:58.026: INFO: PersistentVolume pvc-1fb3e23a-88f4-4267-af34-aeba0686f77f found and phase=Released (107.671686ms) ... skipping 32 lines ... [32m• [SLOW TEST:143.878 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail to use a volume in a pod with mismatched mode [Slow] [90mtest/e2e/storage/testsuites/volumemode.go:299[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":41,"completed":10,"skipped":675,"failed":0} [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumeIO test/e2e/storage/framework/testsuite.go:51 Nov 11 02:45:11.414: INFO: Distro debian doesn't support ntfs -- skipping [AfterEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] volumeIO test/e2e/framework/framework.go:188 ... skipping 346 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with pvc data source in parallel [Slow] [90mtest/e2e/storage/testsuites/provisioning.go:459[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source in parallel [Slow]","total":27,"completed":10,"skipped":993,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m Nov 11 02:46:07.971: INFO: Running AfterSuite actions on all nodes Nov 11 02:46:07.971: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2 Nov 11 02:46:07.971: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2 ... skipping 109 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with pvc data source [90mtest/e2e/storage/testsuites/provisioning.go:421[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source","total":38,"completed":10,"skipped":602,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] provisioning ... skipping 241 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with different volume mode and retain data across pod recreation on the same node [90mtest/e2e/storage/testsuites/multivolume.go:209[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on the same node","total":38,"completed":11,"skipped":896,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral test/e2e/storage/framework/testsuite.go:51 Nov 11 02:46:56.693: INFO: Driver "test.csi.azure.com" does not support volume type "CSIInlineVolume" - skipping ... skipping 24 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (block volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m [90mtest/e2e/storage/testsuites/volumemode.go:299[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 43 lines ... Nov 11 02:45:12.398: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comq58vv] to have phase Bound Nov 11 02:45:12.506: INFO: PersistentVolumeClaim test.csi.azure.comq58vv found but phase is Pending instead of Bound. Nov 11 02:45:14.615: INFO: PersistentVolumeClaim test.csi.azure.comq58vv found but phase is Pending instead of Bound. Nov 11 02:45:16.723: INFO: PersistentVolumeClaim test.csi.azure.comq58vv found and phase=Bound (4.324922236s) [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-6t86 [1mSTEP[0m: Creating a pod to test multi_subpath Nov 11 02:45:17.052: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-6t86" in namespace "provisioning-8993" to be "Succeeded or Failed" Nov 11 02:45:17.162: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 109.339313ms Nov 11 02:45:19.272: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 2.219112808s Nov 11 02:45:21.381: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 4.328486719s Nov 11 02:45:23.489: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43697952s Nov 11 02:45:25.598: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545798581s Nov 11 02:45:27.709: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 10.65659223s ... skipping 25 lines ... Nov 11 02:46:22.557: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 1m5.504570081s Nov 11 02:46:24.666: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.613713204s Nov 11 02:46:26.779: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 1m9.726521029s Nov 11 02:46:28.887: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Pending", Reason="", readiness=false. Elapsed: 1m11.834745993s Nov 11 02:46:30.997: INFO: Pod "pod-subpath-test-dynamicpv-6t86": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m13.944468758s [1mSTEP[0m: Saw pod success Nov 11 02:46:30.997: INFO: Pod "pod-subpath-test-dynamicpv-6t86" satisfied condition "Succeeded or Failed" Nov 11 02:46:31.105: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000001 pod pod-subpath-test-dynamicpv-6t86 container test-container-subpath-dynamicpv-6t86: <nil> [1mSTEP[0m: delete the pod Nov 11 02:46:31.331: INFO: Waiting for pod pod-subpath-test-dynamicpv-6t86 to disappear Nov 11 02:46:31.439: INFO: Pod pod-subpath-test-dynamicpv-6t86 no longer exists [1mSTEP[0m: Deleting pod Nov 11 02:46:31.439: INFO: Deleting pod "pod-subpath-test-dynamicpv-6t86" in namespace "provisioning-8993" ... skipping 21 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should support creating multiple subpath from same volumes [Slow] [90mtest/e2e/storage/testsuites/subpath.go:296[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should support creating multiple subpath from same volumes [Slow]","total":41,"completed":11,"skipped":678,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (ext4)] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:47:13.219: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 31 lines ... [It] should check snapshot fields, check restore correctly works, check deletion (ephemeral) test/e2e/storage/testsuites/snapshottable.go:177 Nov 11 02:44:33.927: INFO: Creating resource for dynamic PV Nov 11 02:44:33.927: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass snapshotting-4372-e2e-scgwfrv [1mSTEP[0m: [init] starting a pod to use the claim Nov 11 02:44:34.146: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-ghnh8" in namespace "snapshotting-4372" to be "Succeeded or Failed" Nov 11 02:44:34.254: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 108.071902ms Nov 11 02:44:36.363: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216632772s Nov 11 02:44:38.472: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326098977s Nov 11 02:44:40.583: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.43637352s Nov 11 02:44:42.692: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.545293875s Nov 11 02:44:44.800: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653902068s ... skipping 5 lines ... Nov 11 02:44:57.455: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 23.308267129s Nov 11 02:44:59.563: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 25.417045225s Nov 11 02:45:01.673: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 27.527021534s Nov 11 02:45:03.783: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Pending", Reason="", readiness=false. Elapsed: 29.63646875s Nov 11 02:45:05.892: INFO: Pod "pvc-snapshottable-tester-ghnh8": Phase="Succeeded", Reason="", readiness=false. Elapsed: 31.745463732s [1mSTEP[0m: Saw pod success Nov 11 02:45:05.892: INFO: Pod "pvc-snapshottable-tester-ghnh8" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim [1mSTEP[0m: creating a SnapshotClass [1mSTEP[0m: creating a dynamic VolumeSnapshot Nov 11 02:45:06.328: INFO: Waiting up to 5m0s for VolumeSnapshot snapshot-lqqrv to become ready Nov 11 02:45:06.437: INFO: VolumeSnapshot snapshot-lqqrv found but is not ready. Nov 11 02:45:08.549: INFO: VolumeSnapshot snapshot-lqqrv found but is not ready. ... skipping 40 lines ... Nov 11 02:46:22.479: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted Nov 11 02:46:23.589: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted Nov 11 02:46:24.699: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted Nov 11 02:46:25.807: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted Nov 11 02:46:26.916: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted Nov 11 02:46:28.025: INFO: volumesnapshotcontents snapcontent-8ca5136b-8b31-4b13-aaa2-702a532783cc has been found and is not deleted Nov 11 02:46:29.025: INFO: WaitUntil failed after reaching the timeout 30s [AfterEach] volume snapshot controller test/e2e/storage/testsuites/snapshottable.go:172 Nov 11 02:46:29.160: INFO: Pod restored-pvc-tester-f8svc has the following logs: Nov 11 02:46:29.160: INFO: Deleting pod "restored-pvc-tester-f8svc" in namespace "snapshotting-4372" Nov 11 02:46:29.270: INFO: Wait up to 5m0s for pod "restored-pvc-tester-f8svc" to be fully deleted Nov 11 02:47:01.486: INFO: deleting snapshot "snapshotting-4372"/"snapshot-lqqrv" ... skipping 31 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works, check deletion (ephemeral) [90mtest/e2e/storage/testsuites/snapshottable.go:177[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Ephemeral Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works, check deletion (ephemeral)","total":28,"completed":11,"skipped":778,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] provisioning[0m [1mshould provision storage with snapshot data source [Feature:VolumeSnapshotDataSource][0m [37mtest/e2e/storage/testsuites/provisioning.go:208[0m ... skipping 126 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] [90mtest/e2e/storage/testsuites/provisioning.go:208[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource]","total":41,"completed":12,"skipped":733,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] provisioning[0m [1mshould provision storage with pvc data source[0m [37mtest/e2e/storage/testsuites/provisioning.go:421[0m ... skipping 103 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] provisioning [90mtest/e2e/storage/framework/testsuite.go:50[0m should provision storage with pvc data source [90mtest/e2e/storage/testsuites/provisioning.go:421[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] provisioning should provision storage with pvc data source","total":38,"completed":12,"skipped":960,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:49:55.395: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:258[0m [36mDistro debian doesn't support ntfs -- skipping[0m test/e2e/storage/framework/testsuite.go:127 [90m------------------------------[0m ... skipping 193 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:168[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":28,"completed":12,"skipped":799,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral[0m [1mshould support two pods which have the same volume definition[0m [37mtest/e2e/storage/testsuites/ephemeral.go:216[0m ... skipping 70 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support two pods which have the same volume definition [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition","total":38,"completed":11,"skipped":714,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:50:51.437: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 108 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should create read/write inline ephemeral volume [90mtest/e2e/storage/testsuites/ephemeral.go:196[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume","total":41,"completed":13,"skipped":815,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] volumes[0m [1mshould store data[0m [37mtest/e2e/storage/testsuites/volumes.go:161[0m ... skipping 108 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should store data [90mtest/e2e/storage/testsuites/volumes.go:161[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] volumes should store data","total":38,"completed":13,"skipped":1012,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m Nov 11 02:51:50.050: INFO: Running AfterSuite actions on all nodes Nov 11 02:51:50.050: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2 Nov 11 02:51:50.050: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2 ... skipping 26 lines ... Nov 11 02:50:29.931: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil Nov 11 02:50:30.041: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.combjp2v] to have phase Bound Nov 11 02:50:30.148: INFO: PersistentVolumeClaim test.csi.azure.combjp2v found but phase is Pending instead of Bound. Nov 11 02:50:32.258: INFO: PersistentVolumeClaim test.csi.azure.combjp2v found but phase is Pending instead of Bound. Nov 11 02:50:34.366: INFO: PersistentVolumeClaim test.csi.azure.combjp2v found and phase=Bound (4.32506118s) [1mSTEP[0m: Creating pod to format volume volume-prep-provisioning-4376 Nov 11 02:50:34.690: INFO: Waiting up to 5m0s for pod "volume-prep-provisioning-4376" in namespace "provisioning-4376" to be "Succeeded or Failed" Nov 11 02:50:34.798: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 107.698244ms Nov 11 02:50:36.907: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216602916s Nov 11 02:50:39.016: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325446657s Nov 11 02:50:41.124: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 6.434378167s Nov 11 02:50:43.237: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 8.547263587s Nov 11 02:50:45.346: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 10.656342627s Nov 11 02:50:47.455: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 12.764506185s Nov 11 02:50:49.564: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 14.873448023s Nov 11 02:50:51.672: INFO: Pod "volume-prep-provisioning-4376": Phase="Pending", Reason="", readiness=false. Elapsed: 16.982095919s Nov 11 02:50:53.783: INFO: Pod "volume-prep-provisioning-4376": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.092585067s [1mSTEP[0m: Saw pod success Nov 11 02:50:53.783: INFO: Pod "volume-prep-provisioning-4376" satisfied condition "Succeeded or Failed" Nov 11 02:50:53.783: INFO: Deleting pod "volume-prep-provisioning-4376" in namespace "provisioning-4376" Nov 11 02:50:53.901: INFO: Wait up to 5m0s for pod "volume-prep-provisioning-4376" to be fully deleted [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-v5fj [1mSTEP[0m: Checking for subpath error in container status Nov 11 02:51:38.336: INFO: Deleting pod "pod-subpath-test-dynamicpv-v5fj" in namespace "provisioning-4376" Nov 11 02:51:38.445: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-v5fj" to be fully deleted [1mSTEP[0m: Deleting pod Nov 11 02:51:40.660: INFO: Deleting pod "pod-subpath-test-dynamicpv-v5fj" in namespace "provisioning-4376" [1mSTEP[0m: Deleting pvc Nov 11 02:51:40.767: INFO: Deleting PersistentVolumeClaim "test.csi.azure.combjp2v" ... skipping 22 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m should verify container cannot write to subpath readonly volumes [Slow] [90mtest/e2e/storage/testsuites/subpath.go:425[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] subPath should verify container cannot write to subpath readonly volumes [Slow]","total":28,"completed":13,"skipped":862,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:52:37.738: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 38 lines ... Nov 11 02:50:52.465: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comh8ftg] to have phase Bound Nov 11 02:50:52.573: INFO: PersistentVolumeClaim test.csi.azure.comh8ftg found but phase is Pending instead of Bound. Nov 11 02:50:54.682: INFO: PersistentVolumeClaim test.csi.azure.comh8ftg found but phase is Pending instead of Bound. Nov 11 02:50:56.791: INFO: PersistentVolumeClaim test.csi.azure.comh8ftg found and phase=Bound (4.326144809s) [1mSTEP[0m: [init] starting a pod to use the claim [1mSTEP[0m: [init] check pod success Nov 11 02:50:57.229: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-ps5w4" in namespace "snapshotting-6604" to be "Succeeded or Failed" Nov 11 02:50:57.338: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 108.013847ms Nov 11 02:50:59.446: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.216514311s Nov 11 02:51:01.555: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.325971556s Nov 11 02:51:03.665: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435216522s Nov 11 02:51:05.774: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.5440876s Nov 11 02:51:07.882: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.652587059s ... skipping 6 lines ... Nov 11 02:51:22.647: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 25.41727561s Nov 11 02:51:24.756: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 27.52655733s Nov 11 02:51:26.865: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 29.635544101s Nov 11 02:51:28.974: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Pending", Reason="", readiness=false. Elapsed: 31.744927803s Nov 11 02:51:31.083: INFO: Pod "pvc-snapshottable-tester-ps5w4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 33.853726164s [1mSTEP[0m: Saw pod success Nov 11 02:51:31.083: INFO: Pod "pvc-snapshottable-tester-ps5w4" satisfied condition "Succeeded or Failed" [1mSTEP[0m: [init] checking the claim Nov 11 02:51:31.192: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.comh8ftg] to have phase Bound Nov 11 02:51:31.300: INFO: PersistentVolumeClaim test.csi.azure.comh8ftg found and phase=Bound (107.756862ms) [1mSTEP[0m: [init] checking the PV [1mSTEP[0m: [init] deleting the pod Nov 11 02:51:31.648: INFO: Pod pvc-snapshottable-tester-ps5w4 has the following logs: ... skipping 35 lines ... Nov 11 02:51:44.478: INFO: WaitUntil finished successfully after 107.99982ms [1mSTEP[0m: getting the snapshot and snapshot content [1mSTEP[0m: checking the snapshot [1mSTEP[0m: checking the SnapshotContent [1mSTEP[0m: Modifying source data test [1mSTEP[0m: modifying the data in the source PVC Nov 11 02:51:45.020: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-data-tester-b97wp" in namespace "snapshotting-6604" to be "Succeeded or Failed" Nov 11 02:51:45.127: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 107.517137ms Nov 11 02:51:47.237: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.217197023s Nov 11 02:51:49.346: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.326059788s Nov 11 02:51:51.456: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.435722444s Nov 11 02:51:53.565: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.544855964s Nov 11 02:51:55.673: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.653123972s ... skipping 12 lines ... Nov 11 02:52:23.096: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 38.075595941s Nov 11 02:52:25.204: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 40.183722572s Nov 11 02:52:27.313: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 42.293008582s Nov 11 02:52:29.423: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Pending", Reason="", readiness=false. Elapsed: 44.402577628s Nov 11 02:52:31.533: INFO: Pod "pvc-snapshottable-data-tester-b97wp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 46.51272373s [1mSTEP[0m: Saw pod success Nov 11 02:52:31.533: INFO: Pod "pvc-snapshottable-data-tester-b97wp" satisfied condition "Succeeded or Failed" Nov 11 02:52:31.753: INFO: Pod pvc-snapshottable-data-tester-b97wp has the following logs: Nov 11 02:52:31.753: INFO: Deleting pod "pvc-snapshottable-data-tester-b97wp" in namespace "snapshotting-6604" Nov 11 02:52:31.867: INFO: Wait up to 5m0s for pod "pvc-snapshottable-data-tester-b97wp" to be fully deleted [1mSTEP[0m: creating a pvc from the snapshot [1mSTEP[0m: starting a pod to use the snapshot Nov 11 02:53:04.412: INFO: Running '/usr/local/bin/kubectl --server=https://kubetest-9juvoppw.westeurope.cloudapp.azure.com --kubeconfig=/root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json --namespace=snapshotting-6604 exec restored-pvc-tester-8s5kn --namespace=snapshotting-6604 -- cat /mnt/test/data' ... skipping 33 lines ... Nov 11 02:53:30.703: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted Nov 11 02:53:31.812: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted Nov 11 02:53:32.921: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted Nov 11 02:53:34.030: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted Nov 11 02:53:35.139: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted Nov 11 02:53:36.248: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted Nov 11 02:53:37.249: INFO: WaitUntil failed after reaching the timeout 30s [AfterEach] volume snapshot controller test/e2e/storage/testsuites/snapshottable.go:172 Nov 11 02:53:37.357: INFO: Error getting logs for pod restored-pvc-tester-8s5kn: the server could not find the requested resource (get pods restored-pvc-tester-8s5kn) Nov 11 02:53:37.357: INFO: Deleting pod "restored-pvc-tester-8s5kn" in namespace "snapshotting-6604" Nov 11 02:53:37.464: INFO: deleting claim "snapshotting-6604"/"pvc-psmfp" Nov 11 02:53:37.572: INFO: deleting snapshot "snapshotting-6604"/"pre-provisioned-snapshot-f6460a03-ec05-4e57-9c9e-3609b18c3cc7" Nov 11 02:53:37.680: INFO: deleting snapshot content "pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7" Nov 11 02:53:38.009: INFO: Waiting up to 5m0s for volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 to be deleted Nov 11 02:53:38.117: INFO: volumesnapshotcontents pre-provisioned-snapcontent-f6460a03-ec05-4e57-9c9e-3609b18c3cc7 has been found and is not deleted ... skipping 27 lines ... [90mtest/e2e/storage/testsuites/snapshottable.go:113[0m [90mtest/e2e/storage/testsuites/snapshottable.go:176[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent) [90mtest/e2e/storage/testsuites/snapshottable.go:278[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion (persistent)","total":38,"completed":12,"skipped":834,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:53:55.645: INFO: Distro debian doesn't support ntfs -- skipping ... skipping 24 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:269[0m [36mDriver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 122 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support two pods which have the same volume definition [90mtest/e2e/storage/testsuites/ephemeral.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition","total":41,"completed":14,"skipped":819,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (filesystem volmode)] multiVolume [Slow] test/e2e/storage/framework/testsuite.go:51 Nov 11 02:54:41.192: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 102 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS] [90mtest/e2e/storage/testsuites/multivolume.go:378[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the volume and its clone from pods on the same node [LinuxOnly][Feature:VolumeSnapshotDataSource][Feature:VolumeSourceXFS]","total":28,"completed":14,"skipped":904,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (ntfs)][Feature:Windows] volumes test/e2e/storage/framework/testsuite.go:51 Nov 11 02:55:04.337: INFO: Driver "test.csi.azure.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 205 lines ... Nov 11 02:54:42.194: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com5dqsl] to have phase Bound Nov 11 02:54:42.302: INFO: PersistentVolumeClaim test.csi.azure.com5dqsl found but phase is Pending instead of Bound. Nov 11 02:54:44.410: INFO: PersistentVolumeClaim test.csi.azure.com5dqsl found but phase is Pending instead of Bound. Nov 11 02:54:46.518: INFO: PersistentVolumeClaim test.csi.azure.com5dqsl found and phase=Bound (4.323904229s) [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-znw6 [1mSTEP[0m: Creating a pod to test exec-volume-test Nov 11 02:54:46.840: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-znw6" in namespace "volume-4145" to be "Succeeded or Failed" Nov 11 02:54:46.948: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 107.284994ms Nov 11 02:54:49.056: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.21599831s Nov 11 02:54:51.164: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.324047074s Nov 11 02:54:53.271: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.430772413s Nov 11 02:54:55.379: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.539219222s Nov 11 02:54:57.487: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.646830909s Nov 11 02:54:59.595: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 12.7545404s Nov 11 02:55:01.703: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 14.862930665s Nov 11 02:55:03.811: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Pending", Reason="", readiness=false. Elapsed: 16.970365321s Nov 11 02:55:05.918: INFO: Pod "exec-volume-test-dynamicpv-znw6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 19.077682475s [1mSTEP[0m: Saw pod success Nov 11 02:55:05.918: INFO: Pod "exec-volume-test-dynamicpv-znw6" satisfied condition "Succeeded or Failed" Nov 11 02:55:06.025: INFO: Trying to get logs from node k8s-agentpool1-30725374-vmss000000 pod exec-volume-test-dynamicpv-znw6 container exec-container-dynamicpv-znw6: <nil> [1mSTEP[0m: delete the pod Nov 11 02:55:06.277: INFO: Waiting for pod exec-volume-test-dynamicpv-znw6 to disappear Nov 11 02:55:06.384: INFO: Pod exec-volume-test-dynamicpv-znw6 no longer exists [1mSTEP[0m: Deleting pod exec-volume-test-dynamicpv-znw6 Nov 11 02:55:06.385: INFO: Deleting pod "exec-volume-test-dynamicpv-znw6" in namespace "volume-4145" ... skipping 21 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] volumes [90mtest/e2e/storage/framework/testsuite.go:50[0m should allow exec of files on the volume [90mtest/e2e/storage/testsuites/volumes.go:198[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume","total":41,"completed":15,"skipped":902,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] volumeMode[0m [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m [37mtest/e2e/storage/testsuites/volumemode.go:299[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (xfs)][Slow] multiVolume [Slow] should concurrently access the single volume from pods on the same node","total":38,"completed":13,"skipped":947,"failed":0} [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode test/e2e/framework/framework.go:187 [1mSTEP[0m: Creating a kubernetes client Nov 11 02:55:36.147: INFO: >>> kubeConfig: /root/tmp3853355860/kubeconfig/kubeconfig.westeurope.json [1mSTEP[0m: Building a namespace api object, basename volumemode [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [1mSTEP[0m: Waiting for kube-root-ca.crt to be provisioned in namespace [It] should fail to use a volume in a pod with mismatched mode [Slow] test/e2e/storage/testsuites/volumemode.go:299 Nov 11 02:55:36.900: INFO: Creating resource for dynamic PV Nov 11 02:55:36.900: INFO: Using claimSize:5Gi, test suite supported size:{ 1Mi}, driver(test.csi.azure.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass volumemode-9653-e2e-scdwfgx [1mSTEP[0m: creating a claim Nov 11 02:55:37.118: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [test.csi.azure.com86cmv] to have phase Bound Nov 11 02:55:37.225: INFO: PersistentVolumeClaim test.csi.azure.com86cmv found but phase is Pending instead of Bound. Nov 11 02:55:39.334: INFO: PersistentVolumeClaim test.csi.azure.com86cmv found but phase is Pending instead of Bound. Nov 11 02:55:41.442: INFO: PersistentVolumeClaim test.csi.azure.com86cmv found and phase=Bound (4.32415817s) [1mSTEP[0m: Creating pod [1mSTEP[0m: Waiting for the pod to fail Nov 11 02:55:44.092: INFO: Deleting pod "pod-6154b4b0-fdc9-4f5a-b3e8-dd37b3524c76" in namespace "volumemode-9653" Nov 11 02:55:44.202: INFO: Wait up to 5m0s for pod "pod-6154b4b0-fdc9-4f5a-b3e8-dd37b3524c76" to be fully deleted [1mSTEP[0m: Deleting pvc Nov 11 02:55:46.420: INFO: Deleting PersistentVolumeClaim "test.csi.azure.com86cmv" Nov 11 02:55:46.529: INFO: Waiting up to 5m0s for PersistentVolume pvc-13c2eca2-4f19-45e4-9e72-66d5b8bc7261 to get deleted Nov 11 02:55:46.640: INFO: PersistentVolume pvc-13c2eca2-4f19-45e4-9e72-66d5b8bc7261 found and phase=Released (110.799251ms) ... skipping 14 lines ... [32m• [SLOW TEST:51.899 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (filesystem volmode)] volumeMode [90mtest/e2e/storage/framework/testsuite.go:50[0m should fail to use a volume in a pod with mismatched mode [Slow] [90mtest/e2e/storage/testsuites/volumemode.go:299[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should fail to use a volume in a pod with mismatched mode [Slow]","total":38,"completed":14,"skipped":947,"failed":0} [36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath test/e2e/storage/framework/testsuite.go:51 Nov 11 02:56:28.050: INFO: Driver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping ... skipping 3 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Inline-volume (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:258[0m [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 8 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: test.csi.azure.com] [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Inline-volume (default fs)] subPath [90mtest/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90mtest/e2e/storage/testsuites/subpath.go:269[0m [36mDriver "test.csi.azure.com" does not support volume type "InlineVolume" - skipping[0m test/e2e/storage/external/external.go:262 [90m------------------------------[0m ... skipping 80 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral [90mtest/e2e/storage/framework/testsuite.go:50[0m should support multiple inline ephemeral volumes [90mtest/e2e/storage/testsuites/ephemeral.go:254[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes","total":41,"completed":16,"skipped":918,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (block volmode)] multiVolume [Slow][0m [1mshould concurrently access the single read-only volume from pods on the same node[0m [37mtest/e2e/storage/testsuites/multivolume.go:423[0m ... skipping 82 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should concurrently access the single read-only volume from pods on the same node [90mtest/e2e/storage/testsuites/multivolume.go:423[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should concurrently access the single read-only volume from pods on the same node","total":41,"completed":17,"skipped":923,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy[0m [1m(Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents[0m [37mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m ... skipping 91 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90mtest/e2e/storage/framework/testsuite.go:50[0m (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [90mtest/e2e/storage/testsuites/fsgroupchangepolicy.go:216[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents","total":41,"completed":18,"skipped":938,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: test.csi.azure.com][0m [90m[Testpattern: Dynamic PV (ext4)] multiVolume [Slow][0m [1mshould access to two volumes with the same volume mode and retain data across pod recreation on different node[0m [37mtest/e2e/storage/testsuites/multivolume.go:168[0m ... skipping 190 lines ... [90mtest/e2e/storage/external/external.go:174[0m [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] [90mtest/e2e/storage/framework/testsuite.go:50[0m should access to two volumes with the same volume mode and retain data across pod recreation on different node [90mtest/e2e/storage/testsuites/multivolume.go:168[0m [90m------------------------------[0m {"msg":"PASSED External Storage [Driver: test.csi.azure.com] [Testpattern: Dynamic PV (ext4)] multiVolume [Slow] should access to two volumes with the same volume mode and retain data across pod recreation on different node","total":41,"completed":19,"skipped":978,"failed":0} [36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m Nov 11 03:02:43.306: INFO: Running AfterSuite actions on all nodes Nov 11 03:02:43.306: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func19.2 Nov 11 03:02:43.306: INFO: Running Cleanup Action: k8s.io/kubernetes/test/e2e/storage.glob..func9.2 ... skipping 15 lines ... Nov 11 03:02:43.339: INFO: Running AfterSuite actions on node 1 [91m[1mSummarizing 1 Failure:[0m [91m[1m[Fail] [0m[90mExternal Storage [Driver: test.csi.azure.com] [0m[0m[Testpattern: Dynamic PV (block volmode)] volumeMode [0m[91m[1m[Measurement] should not mount / map unused volumes in a pod [LinuxOnly] [0m [37mtest/e2e/storage/testsuites/volumemode.go:190[0m [1m[91mRan 89 of 7227 Specs in 3341.042 seconds[0m [1m[91mFAIL![0m -- [32m[1m88 Passed[0m | [91m[1m1 Failed[0m | [33m[1m0 Pending[0m | [36m[1m7138 Skipped[0m Ginkgo ran 1 suite in 55m44.542035019s Test Suite Failed + print_logs + sed -i s/disk.csi.azure.com/test.csi.azure.com/g deploy/example/storageclass-azuredisk-csi.yaml + bash ./hack/verify-examples.sh linux azurepubliccloud ephemeral test begin to create deployment examples ... storageclass.storage.k8s.io/managed-csi created Applying config "deploy/example/deployment.yaml" ... skipping 102 lines ... I1111 02:06:52.481911 1 reflector.go:257] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:134 I1111 02:06:52.482015 1 reflector.go:221] Starting reflector *v1.CustomResourceDefinition (30s) from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117 I1111 02:06:52.482033 1 reflector.go:257] Listing and watching *v1.CustomResourceDefinition from k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117 I1111 02:06:52.582045 1 shared_informer.go:285] caches populated I1111 02:06:52.582088 1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 e2e-test I1111 02:06:52.582120 1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider I1111 02:06:52.584942 1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1111 02:06:52.584962 1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider I1111 02:06:52.584967 1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1111 02:06:52.584996 1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully I1111 02:06:52.585742 1 azure_auth.go:245] Using AzurePublicCloud environment I1111 02:06:52.585765 1 azure_auth.go:96] azure: using managed identity extension to retrieve access token I1111 02:06:52.585769 1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token ... skipping 29 lines ... I1111 02:06:52.586164 1 azure_vmasclient.go:73] Azure AvailabilitySetsClient (write ops) using rate limit config: QPS=100, bucket=1000 I1111 02:06:52.586226 1 azure.go:1039] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333000, Bucket: 240 I1111 02:06:52.586272 1 azuredisk_v2.go:215] disable UseInstanceMetadata for controller I1111 02:06:52.586286 1 azuredisk_v2.go:225] DisableAvailabilitySetNodes for controller since current VMType is vmss I1111 02:06:52.586292 1 azuredisk_v2.go:231] cloud: AzurePublicCloud, location: westeurope, rg: kubetest-9juvoppw, VMType: vmss, PrimaryScaleSetName: k8s-agentpool1-30725374-vmss, PrimaryAvailabilitySetName: , DisableAvailabilitySetNodes: true I1111 02:06:52.586298 1 skus.go:123] NewNodeInfo: Starting to populate node and disk sku information. E1111 02:06:52.586313 1 azuredisk_v2.go:239] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance I1111 02:06:52.586418 1 mount_linux.go:208] Detected OS without systemd I1111 02:06:52.586432 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME I1111 02:06:52.586438 1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME I1111 02:06:52.586443 1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I1111 02:06:52.586448 1 driver.go:81] Enabling controller service capability: CLONE_VOLUME I1111 02:06:52.586453 1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME ... skipping 273 lines ... I1111 02:06:52.756016 1 shared_state.go:487] "msg"="Storing pod cloud-node-manager-fqh5j and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" I1111 02:06:52.756032 1 pod.go:91] "msg"="Creating replicas for pod kube-system/cloud-node-manager-fqh5j." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-fqh5j" I1111 02:06:52.756041 1 shared_state.go:311] "msg"="Getting requested volumes for pod (kube-system/cloud-node-manager-fqh5j)." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-fqh5j" I1111 02:06:52.756051 1 pod.go:99] "msg"="Pod kube-system/cloud-node-manager-fqh5j has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-fqh5j" I1111 02:06:52.756069 1 pod.go:89] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/cloud-node-manager-fqh5j" "latency"=36201 I1111 02:06:52.756081 1 shared_state.go:417] "msg"="Adding pod csi-test-controller-78545b9dc-fqs6m to shared map with keyName kube-system/csi-test-controller-78545b9dc-fqs6m." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" I1111 02:06:52.757693 1 shared_state.go:423] "msg"="Pod spec of pod csi-test-controller-78545b9dc-fqs6m is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-k892x VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=30s --leader-election --leader-election-namespace=kube-system --worker-threads=100 --extra-create-metadata=true --strict-topology=true --kube-api-qps=50 --kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500 -kube-api-qps=50 -kube-api-burst=100] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 Command:[] Args:[--v=5 --config=/etc/csi-test-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-test-controller-config ReadOnly:false MountPath:/etc/csi-test-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-k892x ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000d6d190 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-agentpool1-30725374-vmss000001 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000d6d198} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000d6d1a0}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000d6d1a8 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000d6d1ac PreemptionPolicy:0xc000946040 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil>}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-test-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-k892x VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" I1111 02:06:52.757994 1 shared_state.go:461] "msg"="Pod csi-test-controller-78545b9dc-fqs6m: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" I1111 02:06:52.758046 1 shared_state.go:461] "msg"="Pod csi-test-controller-78545b9dc-fqs6m: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" I1111 02:06:52.758082 1 shared_state.go:461] "msg"="Pod csi-test-controller-78545b9dc-fqs6m: Skipping Volume {csi-test-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-test-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" I1111 02:06:52.758139 1 shared_state.go:461] "msg"="Pod csi-test-controller-78545b9dc-fqs6m: Skipping Volume {kube-api-access-k892x {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" I1111 02:06:52.758161 1 shared_state.go:487] "msg"="Storing pod csi-test-controller-78545b9dc-fqs6m and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" I1111 02:06:52.758183 1 pod.go:91] "msg"="Creating replicas for pod kube-system/csi-test-controller-78545b9dc-fqs6m." "disk.csi.azure.com/request-id"="827bea31-6165-11ed-a60b-6045bd88ebdb" "disk.csi/azure.com/pod-name"="kube-system/csi-test-controller-78545b9dc-fqs6m" ... skipping 140 lines ... I1111 02:06:52.811498 1 shared_informer.go:285] caches populated I1111 02:06:52.811515 1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="azvolumeattachment" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10 I1111 02:06:52.811522 1 shared_informer.go:285] caches populated I1111 02:06:52.811537 1 shared_informer.go:285] caches populated I1111 02:06:52.811539 1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="azvolume" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10 I1111 02:06:52.811545 1 shared_informer.go:285] caches populated I1111 02:06:52.811620 1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I1111 02:06:52.811644 1 shared_informer.go:285] caches populated I1111 02:06:52.811662 1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="pod" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10 I1111 02:06:52.811666 1 shared_state.go:1068] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="828b5ebb-6165-11ed-a60b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8500 I1111 02:06:52.811674 1 shared_informer.go:285] caches populated I1111 02:06:52.811697 1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="volumeattachment" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10 I1111 02:06:52.811620 1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I1111 02:06:52.811708 1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I1111 02:06:52.811729 1 shared_informer.go:285] caches populated I1111 02:06:52.811747 1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-master-30725374-0" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" I1111 02:06:52.811761 1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="replica" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10 I1111 02:06:52.811759 1 shared_state.go:1068] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="828b61ef-6165-11ed-a60b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=11600 I1111 02:06:52.811781 1 shared_state.go:1068] "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="828b635d-6165-11ed-a60b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=6800 I1111 02:06:52.811802 1 shared_state.go:417] "msg"="Adding pod coredns-7cf78b68d8-zd9qd to shared map with keyName kube-system/coredns-7cf78b68d8-zd9qd." I1111 02:06:52.811813 1 shared_state.go:417] "msg"="Adding pod csi-test-scheduler-extender-748844575-n8kpw to shared map with keyName kube-system/csi-test-scheduler-extender-748844575-n8kpw." I1111 02:06:52.811847 1 shared_state.go:417] "msg"="Adding pod csi-test-node-lvncc to shared map with keyName kube-system/csi-test-node-lvncc." ... skipping 34586 lines ... I1111 02:06:47.667021 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2556&timeout=9m52s&timeoutSeconds=592&watch=true 200 OK in 0 milliseconds I1111 02:06:47.667839 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2332&timeout=5m24s&timeoutSeconds=324&watch=true 200 OK in 0 milliseconds I1111 02:06:47.760082 1 shared_informer.go:285] caches populated I1111 02:06:47.760132 1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 e2e-test I1111 02:06:47.760157 1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider I1111 02:06:47.763170 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds I1111 02:06:47.763407 1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1111 02:06:47.763424 1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider I1111 02:06:47.763433 1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1111 02:06:47.763453 1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully I1111 02:06:47.764187 1 azure_auth.go:245] Using AzurePublicCloud environment I1111 02:06:47.764207 1 azure_auth.go:96] azure: using managed identity extension to retrieve access token I1111 02:06:47.764213 1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token ... skipping 53 lines ... I1111 02:06:47.888025 1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes 201 Created in 4 milliseconds I1111 02:06:47.892113 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 3 milliseconds I1111 02:06:47.892271 1 azuredisk_v2.go:538] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7f9b50c0-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).registerAzDriverNode" "latency"=9262490 I1111 02:06:47.892363 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:06:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7f9cbd7c-6165-11ed-ae2b-6045bd88ebdb" I1111 02:06:47.892507 1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"} I1111 02:06:47.897506 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 409 Conflict in 5 milliseconds E1111 02:06:47.897629 1 azuredisk_v2.go:597] "msg"="Failed to update AzDriverNode status after creation" "error"="Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-30725374-vmss000001\": the object has been modified; please apply your changes to the latest version and try again" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7f9cbd7c-6165-11ed-ae2b-6045bd88ebdb" E1111 02:06:47.897678 1 azuredisk_v2.go:549] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [Operation cannot be fulfilled on azdrivernodes.disk.csi.azure.com \"k8s-agentpool1-30725374-vmss000001\": the object has been modified; please apply your changes to the latest version and try again]" "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7f9cbd7c-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5335152 I1111 02:06:47.897687 1 azuredisk_v2.go:512] Starting heartbeat loop with initial delay 29.777s and frequency 30s I1111 02:06:48.316868 1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo I1111 02:06:48.316891 1 utils.go:79] GRPC request: {} I1111 02:06:48.319115 1 utils.go:85] GRPC response: {"name":"test.csi.azure.com","vendor_version":"latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0"} I1111 02:06:48.782494 1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo I1111 02:06:48.782517 1 utils.go:79] GRPC request: {} ... skipping 36 lines ... I1111 02:07:37.131112 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:07:37.131132 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3af14f39-2800-4b53-9da0-715b511c77c6","csi.storage.k8s.io/pvc/name":"test.csi.azure.com7t57r","csi.storage.k8s.io/pvc/namespace":"snapshotting-4409","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3af14f39-2800-4b53-9da0-715b511c77c6"} I1111 02:07:37.131310 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3af14f39-2800-4b53-9da0-715b511c77c6-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:07:47.595458 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:47.595591 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:47.595600 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:47.595866 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:07:47.595907 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:07:47.668259 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:47.675498 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:07:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a33eeaf2-6165-11ed-ae2b-6045bd88ebdb" I1111 02:07:47.682507 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:07:47.682719 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a33eeaf2-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7279867 I1111 02:07:47.786685 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:59.199074 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:07:59.199100 1 utils.go:79] GRPC request: {} I1111 02:07:59.199145 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:08:17.596366 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:17.596400 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:17.596384 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:17.596514 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:08:17.596550 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:08:17.668971 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:17.676239 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:08:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b520ac50-6165-11ed-ae2b-6045bd88ebdb" I1111 02:08:17.685580 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 9 milliseconds I1111 02:08:17.685720 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b520ac50-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9506393 I1111 02:08:17.786997 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:29.197595 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:08:29.197616 1 utils.go:79] GRPC request: {} I1111 02:08:29.197659 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:08:47.597083 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:47.597181 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:47.597211 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:47.597246 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:08:47.597275 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:08:47.669150 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:47.676440 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:08:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c7025692-6165-11ed-ae2b-6045bd88ebdb" I1111 02:08:47.681963 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:08:47.682177 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c7025692-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5766168 I1111 02:08:47.787921 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:59.197420 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:08:59.197443 1 utils.go:79] GRPC request: {} I1111 02:08:59.197488 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:09:17.597963 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:17.598078 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:09:17.598096 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:17.598120 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:17.598110 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:09:17.669470 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:17.675749 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:09:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d8e3deef-6165-11ed-ae2b-6045bd88ebdb" I1111 02:09:17.683035 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds I1111 02:09:17.683212 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d8e3deef-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7493859 I1111 02:09:17.788274 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:29.197530 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:09:29.197552 1 utils.go:79] GRPC request: {} I1111 02:09:29.197588 1 utils.go:85] GRPC response: {"ready":{"value":true}} E1111 02:09:36.725816 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9cb8bd86-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e10ddcaf-7585-41ea-b3e6-4403705103b6" "latency"=119995831254 E1111 02:09:36.725890 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9cb8bd86-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119996109556 E1111 02:09:36.726080 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e10ddcaf-7585-41ea-b3e6-4403705103b6) to be attached to node (k8s-agentpool1-30725374-vmss000001): context canceled E1111 02:09:37.126712 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9cf6029b-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3af14f39-2800-4b53-9da0-715b511c77c6" "latency"=119995297584 E1111 02:09:37.126781 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9cf6029b-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119995482285 E1111 02:09:37.126803 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3af14f39-2800-4b53-9da0-715b511c77c6) to be attached to node (k8s-agentpool1-30725374-vmss000001): context canceled I1111 02:09:37.285429 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:09:37.285449 1 utils.go:79] GRPC request: {} I1111 02:09:37.285483 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:09:37.288612 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:09:37.288633 1 utils.go:79] GRPC request: {} I1111 02:09:37.288664 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 15 lines ... I1111 02:09:37.692077 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:09:37.692091 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3af14f39-2800-4b53-9da0-715b511c77c6","csi.storage.k8s.io/pvc/name":"test.csi.azure.com7t57r","csi.storage.k8s.io/pvc/namespace":"snapshotting-4409","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3af14f39-2800-4b53-9da0-715b511c77c6"} I1111 02:09:37.692262 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3af14f39-2800-4b53-9da0-715b511c77c6-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:09:47.598982 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:47.599042 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:47.599093 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:47.599181 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:09:47.599202 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:09:47.670520 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:47.675809 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:09:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="eac583a6-6165-11ed-ae2b-6045bd88ebdb" I1111 02:09:47.681810 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:09:47.682312 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="eac583a6-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6500957 I1111 02:09:47.789186 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:59.197387 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:09:59.197409 1 utils.go:79] GRPC request: {} I1111 02:09:59.197447 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:10:17.599352 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:17.599374 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:17.599399 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:17.599459 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:10:17.599569 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:10:17.671060 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:17.676330 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:10:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="fca73b9d-6165-11ed-ae2b-6045bd88ebdb" I1111 02:10:17.683059 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:10:17.683198 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="fca73b9d-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6896860 I1111 02:10:17.789292 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:29.197744 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:10:29.197769 1 utils.go:79] GRPC request: {} I1111 02:10:29.197805 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:10:47.600202 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:47.600225 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:47.600258 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:47.600367 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:10:47.600387 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:10:47.671838 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:47.676131 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:10:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0e88d62b-6166-11ed-ae2b-6045bd88ebdb" I1111 02:10:47.681681 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:10:47.682506 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0e88d62b-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6410856 I1111 02:10:47.789716 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:59.197178 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:10:59.197211 1 utils.go:79] GRPC request: {} I1111 02:10:59.197252 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:11:17.601231 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:17.601312 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:17.601391 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:17.601418 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:11:17.601432 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:11:17.672843 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:17.676121 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:11:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="206a798e-6166-11ed-ae2b-6045bd88ebdb" I1111 02:11:17.681295 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:11:17.681851 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="206a798e-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5758349 I1111 02:11:17.790821 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:29.197129 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:11:29.197151 1 utils.go:79] GRPC request: {} I1111 02:11:29.197188 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:11:32.813354 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:11:32.813404 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e4d226d8-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3af14f39-2800-4b53-9da0-715b511c77c6" "latency"=115121091923 I1111 02:11:32.813426 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e4d226d8-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=115121182623 I1111 02:11:32.815896 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:11:32.815940 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e494d466-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e10ddcaf-7585-41ea-b3e6-4403705103b6" "latency"=115525504766 I1111 02:11:32.815963 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e494d466-6165-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=115525598166 I1111 02:11:33.403235 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/ I1111 02:11:33.403251 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/ I1111 02:11:33.403281 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:11:33.403282 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType ... skipping 181 lines ... I1111 02:12:10.089312 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3af14f39-2800-4b53-9da0-715b511c77c6-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:12:11.668738 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: Watch close - *v1.CustomResourceDefinition total 0 items received I1111 02:12:11.672407 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2332&timeout=9m38s&timeoutSeconds=578&watch=true 200 OK in 3 milliseconds I1111 02:12:17.602594 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:17.602631 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:17.602683 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:17.602722 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:12:17.674168 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:17.676391 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:12:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="442dc998-6166-11ed-ae2b-6045bd88ebdb" I1111 02:12:17.682156 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:12:17.682375 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="442dc998-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6022248 I1111 02:12:17.792194 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:29.197398 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:12:29.197420 1 utils.go:79] GRPC request: {} I1111 02:12:29.197458 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:12:39.596252 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 55 items received I1111 02:12:39.598858 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=4577&timeout=8m39s&timeoutSeconds=519&watch=true 200 OK in 2 milliseconds I1111 02:12:39.922853 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:12:39.922915 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3fa8188f-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3af14f39-2800-4b53-9da0-715b511c77c6" "latency"=29833545015 I1111 02:12:39.922946 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3fa8188f-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29833645915 I1111 02:12:40.545135 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I1111 02:12:40.545191 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:12:40.545223 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount with mount options([]) I1111 02:12:40.545236 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 55 lines ... W1111 02:12:43.439835 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount" is not a mountpoint, deleting I1111 02:12:43.439894 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3af14f39-2800-4b53-9da0-715b511c77c6/globalmount successfully I1111 02:12:43.439903 1 utils.go:85] GRPC response: {} I1111 02:12:47.603400 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:47.603487 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:47.603485 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:47.603581 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:12:47.675033 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:47.676196 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:12:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="560f64e3-6166-11ed-ae2b-6045bd88ebdb" I1111 02:12:47.683516 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds I1111 02:12:47.683753 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="560f64e3-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7595360 I1111 02:12:47.792873 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:59.197996 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:12:59.198015 1 utils.go:79] GRPC request: {} I1111 02:12:59.198052 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:13:17.604191 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:17.604285 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:17.604314 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:17.604391 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:13:17.675937 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:17.676059 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:13:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="67f10289-6166-11ed-ae2b-6045bd88ebdb" I1111 02:13:17.694432 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 18 milliseconds I1111 02:13:17.694644 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="67f10289-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18590845 I1111 02:13:17.793281 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:29.198146 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:13:29.198173 1 utils.go:79] GRPC request: {} I1111 02:13:29.198219 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:13:30.983498 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:13:30.983571 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="52c09510-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582" "latency"=48856998667 I1111 02:13:30.983601 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="52c09510-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=48857119468 I1111 02:13:31.499712 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:13:31.499732 1 utils.go:79] GRPC request: {} I1111 02:13:31.499767 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:13:31.502614 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 65 lines ... I1111 02:13:43.988863 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount I1111 02:13:43.988890 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount" is a mountpoint, unmounting I1111 02:13:43.988903 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount W1111 02:13:44.022875 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount" is not a mountpoint, deleting I1111 02:13:44.022953 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ee8f9fc-8188-4d77-ba26-dfc4169f7582/globalmount successfully I1111 02:13:44.022965 1 utils.go:85] GRPC response: {} I1111 02:13:46.447704 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:13:46.447769 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="702f0db2-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6f481851-34b4-4af1-9f4f-03d165ba9b82" "latency"=14943298593 I1111 02:13:46.447808 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="702f0db2-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14943414495 I1111 02:13:46.451876 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:13:46.451944 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="71810849-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-008fe7f7-5797-41ca-addb-b2e8a842b067" "latency"=12732488065 I1111 02:13:46.451974 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="71810849-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12732607166 I1111 02:13:47.065882 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/ I1111 02:13:47.065932 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ I1111 02:13:47.065969 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:13:47.066018 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6f481851-34b4-4af1-9f4f-03d165ba9b82/globalmount with mount options([]) ... skipping 204 lines ... I1111 02:14:59.197218 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:14:59.197244 1 utils.go:79] GRPC request: {} I1111 02:14:59.197285 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:15:17.607023 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:17.607059 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:17.607151 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:17.607215 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:15:17.675481 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:15:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="af77784a-6166-11ed-ae2b-6045bd88ebdb" I1111 02:15:17.678805 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:17.681218 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:15:17.681421 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="af77784a-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5970848 I1111 02:15:17.795721 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:29.197507 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:15:29.197530 1 utils.go:79] GRPC request: {} I1111 02:15:29.197571 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:15:47.607906 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:47.608061 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:47.608084 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:47.608161 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:15:47.675518 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:15:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c1591cf7-6166-11ed-ae2b-6045bd88ebdb" I1111 02:15:47.679215 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:47.681296 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:15:47.681937 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c1591cf7-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6447950 I1111 02:15:47.796843 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:15:59.197076 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:15:59.197109 1 utils.go:79] GRPC request: {} I1111 02:15:59.197176 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:16:01.397375 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:16:01.397436 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a1b50433-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-644fc064-7724-4609-ade0-52427c92efcb" "latency"=66806668987 I1111 02:16:01.397460 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a1b50433-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66806773689 I1111 02:16:03.110174 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I1111 02:16:03.110214 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:16:03.110241 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-644fc064-7724-4609-ade0-52427c92efcb/globalmount with mount options([]) I1111 02:16:03.110249 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 58 lines ... I1111 02:16:09.427493 1 utils.go:85] GRPC response: {} I1111 02:16:12.596579 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 210 items received I1111 02:16:12.599499 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=6769&timeout=9m50s&timeoutSeconds=590&watch=true 200 OK in 2 milliseconds I1111 02:16:17.608547 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:17.608628 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:17.608537 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:17.608841 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:16:17.676399 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:16:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d33ae121-6166-11ed-ae2b-6045bd88ebdb" I1111 02:16:17.679419 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:17.681191 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds I1111 02:16:17.681413 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d33ae121-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5075940 I1111 02:16:17.797558 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:29.198086 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:16:29.198115 1 utils.go:79] GRPC request: {} I1111 02:16:29.198163 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:16:32.014304 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:16:32.014361 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="cbc8c5a1-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-822db309-9767-4898-b4da-11a98999ff89" "latency"=26829800152 I1111 02:16:32.014381 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cbc8c5a1-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26829919253 I1111 02:16:32.380456 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:16:32.380477 1 utils.go:79] GRPC request: {} I1111 02:16:32.380512 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:16:32.381385 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 55 lines ... I1111 02:16:38.748063 1 utils.go:85] GRPC response: {} I1111 02:16:39.668167 1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 102 items received I1111 02:16:39.670702 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=7036&timeout=8m9s&timeoutSeconds=489&watch=true 200 OK in 2 milliseconds I1111 02:16:47.609736 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:47.609817 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:47.609831 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:47.609897 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:16:47.676265 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:16:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e51c7f79-6166-11ed-ae2b-6045bd88ebdb" I1111 02:16:47.680172 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:47.681699 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:16:47.681912 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e51c7f79-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5687244 I1111 02:16:47.798146 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:16:59.197386 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:16:59.197410 1 utils.go:79] GRPC request: {} I1111 02:16:59.197451 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:17:17.610017 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:17.610049 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:17.610088 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:17.610153 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:17:17.676484 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:17:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f6fe2ace-6166-11ed-ae2b-6045bd88ebdb" I1111 02:17:17.681042 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:17.681584 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds I1111 02:17:17.681757 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f6fe2ace-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5327042 I1111 02:17:17.798345 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:29.197602 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:17:29.197622 1 utils.go:79] GRPC request: {} I1111 02:17:29.197656 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:17:47.610160 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:47.610262 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:47.610280 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:17:47.610297 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:47.675615 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:17:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="08dfab7a-6167-11ed-ae2b-6045bd88ebdb" I1111 02:17:47.681903 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:47.682078 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:17:47.682285 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="08dfab7a-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6730852 I1111 02:17:47.799416 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:17:58.664418 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:17:58.664492 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="dbfedce3-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-955327db-3ec7-467b-8608-a885eb36800a" "latency"=86281914489 I1111 02:17:58.664516 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="dbfedce3-6166-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=86282017389 I1111 02:17:59.174060 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:17:59.174089 1 utils.go:79] GRPC request: {} I1111 02:17:59.174137 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:17:59.177997 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 42 lines ... I1111 02:18:07.402929 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I1111 02:18:07.402944 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-955327db-3ec7-467b-8608-a885eb36800a","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-955327db-3ec7-467b-8608-a885eb36800a/dev/4534e6eb-4029-4bc9-bdd7-9381e077f6a1"} I1111 02:18:07.403631 1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]} I1111 02:18:17.610661 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:17.610674 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:17.610684 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:17.610889 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:18:17.610922 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:18:17.675962 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:18:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1ac15ca5-6167-11ed-ae2b-6045bd88ebdb" I1111 02:18:17.682811 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:18:17.682834 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:17.684585 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1ac15ca5-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8661568 I1111 02:18:17.799668 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:24.144708 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:18:24.144766 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0fbb34cf-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8b06a43e-3ded-4b4c-8798-35687a3c310e" "latency"=24964068568 I1111 02:18:24.144795 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0fbb34cf-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24964161368 I1111 02:18:24.148508 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:18:24.148568 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1026d4f1-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-adf86ba4-0150-4820-bc17-b65dadcd974f" "latency"=24262529739 I1111 02:18:24.148596 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1026d4f1-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24262634041 I1111 02:18:24.739652 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 02:18:24.739672 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/ I1111 02:18:24.739700 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:18:24.739700 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType ... skipping 126 lines ... W1111 02:18:37.972033 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-955327db-3ec7-467b-8608-a885eb36800a" is not a mountpoint, deleting I1111 02:18:37.972112 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-955327db-3ec7-467b-8608-a885eb36800a successfully I1111 02:18:37.972126 1 utils.go:85] GRPC response: {} I1111 02:18:47.611234 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:47.611277 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:47.611334 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:47.611398 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:18:47.675729 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:18:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2ca2f63d-6167-11ed-ae2b-6045bd88ebdb" I1111 02:18:47.681983 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:18:47.682222 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2ca2f63d-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6535851 I1111 02:18:47.683356 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:47.800684 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:59.198001 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:18:59.198029 1 utils.go:79] GRPC request: {} I1111 02:18:59.198078 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:19:00.193051 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:19:00.193120 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="21d155b0-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3a8f8efe-d680-482a-bd4a-8b620166b0c6" "latency"=30668404491 I1111 02:19:00.193149 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="21d155b0-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30668507992 I1111 02:19:00.790570 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/ I1111 02:19:00.790623 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I1111 02:19:00.790652 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3a8f8efe-d680-482a-bd4a-8b620166b0c6/globalmount with mount options([]) I1111 02:19:00.790663 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3]) ... skipping 66 lines ... I1111 02:19:20.825920 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-48a25eef-bd2b-459f-911a-136e5183b043-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:19:29.197121 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:19:29.197144 1 utils.go:79] GRPC request: {} I1111 02:19:29.197179 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:19:47.612491 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:19:47.612506 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:19:47.612610 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:19:47.612518 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:19:47.675994 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:19:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5066478c-6167-11ed-ae2b-6045bd88ebdb" I1111 02:19:47.682049 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:19:47.682714 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5066478c-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6759137 I1111 02:19:47.684232 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:19:47.801550 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:19:59.197957 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:19:59.197987 1 utils.go:79] GRPC request: {} I1111 02:19:59.198030 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:20:01.453183 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:20:01.453259 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="40654957-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-48a25eef-bd2b-459f-911a-136e5183b043" "latency"=40627263447 I1111 02:20:01.453304 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="40654957-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40627388649 I1111 02:20:01.921933 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:20:01.921961 1 utils.go:79] GRPC request: {} I1111 02:20:01.922008 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:20:01.926380 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 42 lines ... I1111 02:20:03.491237 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount","target_path":"/var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-48a25eef-bd2b-459f-911a-136e5183b043","csi.storage.k8s.io/pvc/name":"test.csi.azure.comfptp8","csi.storage.k8s.io/pvc/namespace":"volume-9791","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-48a25eef-bd2b-459f-911a-136e5183b043"} I1111 02:20:03.491392 1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount at /var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount I1111 02:20:03.491406 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount /var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount) I1111 02:20:03.492255 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount /var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount) I1111 02:20:03.493157 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-48a25eef-bd2b-459f-911a-136e5183b043/globalmount at /var/lib/kubelet/pods/4ebf2513-33d4-4c65-bbc6-f84c21b4b53b/volumes/kubernetes.io~csi/pvc-48a25eef-bd2b-459f-911a-136e5183b043/mount successfully I1111 02:20:03.493180 1 utils.go:85] GRPC response: {} I1111 02:20:11.719580 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:20:11.719634 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="58e5281e-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-50522e90-872b-46df-8acf-7267e686b23e" "latency"=9790326818 I1111 02:20:11.719665 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="58e5281e-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9790428619 I1111 02:20:11.720962 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:20:11.721036 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5902fbb9-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-874c6d1e-232f-4c47-b057-d06057d49f52" "latency"=9596239326 I1111 02:20:11.721063 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5902fbb9-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9596361827 I1111 02:20:12.314211 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/ I1111 02:20:12.314327 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ I1111 02:20:12.314373 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:20:12.314423 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-50522e90-872b-46df-8acf-7267e686b23e/globalmount with mount options([]) ... skipping 161 lines ... I1111 02:20:37.063948 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I1111 02:20:37.063963 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-874c6d1e-232f-4c47-b057-d06057d49f52","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-874c6d1e-232f-4c47-b057-d06057d49f52/dev/20f9a18f-690a-4809-9181-abc75739c042"} I1111 02:20:37.065262 1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]} I1111 02:20:47.613997 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:20:47.614063 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:20:47.614071 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:20:47.614181 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:20:47.614227 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:20:47.675613 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:20:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="74297b89-6167-11ed-ae2b-6045bd88ebdb" I1111 02:20:47.682098 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:20:47.682707 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="74297b89-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7203857 I1111 02:20:47.684591 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:20:47.802860 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:20:59.198891 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:20:59.198918 1 utils.go:79] GRPC request: {} I1111 02:20:59.198968 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:21:17.614708 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:21:17.614750 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:21:17.614717 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:21:17.614828 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:21:17.614844 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:21:17.676073 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:21:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="860b32e4-6167-11ed-ae2b-6045bd88ebdb" I1111 02:21:17.681621 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:21:17.681871 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="860b32e4-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5833344 I1111 02:21:17.684908 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:21:17.803228 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:21:18.599155 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 80 items received I1111 02:21:18.602927 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=9739&timeout=6m1s&timeoutSeconds=361&watch=true 200 OK in 3 milliseconds I1111 02:21:29.197364 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:21:29.197391 1 utils.go:79] GRPC request: {} I1111 02:21:29.197439 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:21:29.499738 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:21:29.499786 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="6a9c78a1-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b61cfee1-08a3-45b0-a959-f7416124b164" "latency"=57847858537 I1111 02:21:29.499806 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6a9c78a1-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=57847940239 I1111 02:21:29.500246 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:21:29.500323 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="6a9c754f-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d4da29a0-a8d5-47e6-a4bb-af8d60bcc7e6" "latency"=57848428742 I1111 02:21:29.500355 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="6a9c754f-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=57848568543 I1111 02:21:30.107279 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdc under /dev/disk/azure/scsi1/ I1111 02:21:30.107282 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdd under /dev/disk/azure/scsi1/ I1111 02:21:30.107332 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType I1111 02:21:30.107354 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType ... skipping 75 lines ... I1111 02:21:30.607536 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount","target_path":"/var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b61cfee1-08a3-45b0-a959-f7416124b164","csi.storage.k8s.io/pvc/name":"test.csi.azure.comw27rm","csi.storage.k8s.io/pvc/namespace":"multivolume-6915","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-b61cfee1-08a3-45b0-a959-f7416124b164"} I1111 02:21:30.607711 1 nodeserver_v2.go:348] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount at /var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount I1111 02:21:30.607728 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount /var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount) I1111 02:21:30.608486 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount /var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount) I1111 02:21:30.609300 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/globalmount at /var/lib/kubelet/pods/c48844af-3f59-4185-a183-9434c8cbf6df/volumes/kubernetes.io~csi/pvc-b61cfee1-08a3-45b0-a959-f7416124b164/mount successfully I1111 02:21:30.609316 1 utils.go:85] GRPC response: {} I1111 02:21:39.847584 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:21:39.847653 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8d976b8c-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-df212b9f-8dc2-4e69-92ad-d118022abf03" "latency"=9508560846 I1111 02:21:39.847683 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8d976b8c-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9508657348 I1111 02:21:39.849920 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:21:39.849994 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8d9734b6-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7fece55f-2db4-48a2-acbc-d5a038b3fb09" "latency"=9512283975 I1111 02:21:39.850031 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8d9734b6-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9512408976 I1111 02:21:40.476973 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdg under /dev/disk/azure/scsi1/ I1111 02:21:40.477051 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:21:40.477066 1 utils.go:85] GRPC response: {} I1111 02:21:40.477185 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdf under /dev/disk/azure/scsi1/ ... skipping 278 lines ... I1111 02:22:13.147633 1 utils.go:85] GRPC response: {} W1111 02:22:13.153641 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d4da29a0-a8d5-47e6-a4bb-af8d60bcc7e6/globalmount" is not a mountpoint, deleting I1111 02:22:13.153699 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d4da29a0-a8d5-47e6-a4bb-af8d60bcc7e6/globalmount successfully I1111 02:22:13.153713 1 utils.go:85] GRPC response: {} I1111 02:22:17.615810 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:17.615930 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:17.615971 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:22:17.616004 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:17.676353 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:22:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a9ce833b-6167-11ed-ae2b-6045bd88ebdb" I1111 02:22:17.683818 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds I1111 02:22:17.684035 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a9ce833b-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7739167 I1111 02:22:17.685990 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:17.804505 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:22.790270 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 25 items received I1111 02:22:22.791495 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000001&resourceVersion=10328&timeout=7m6s&timeoutSeconds=426&watch=true 200 OK in 1 milliseconds I1111 02:22:29.198073 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:22:29.198099 1 utils.go:79] GRPC request: {} I1111 02:22:29.198142 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:22:35.278352 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:22:35.278425 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a18354b2-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0a32ef96-6d0c-415d-9fff-f5f376d27c96" "latency"=31516533271 I1111 02:22:35.278453 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a18354b2-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31516654372 I1111 02:22:35.867803 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sde under /dev/disk/azure/scsi1/ I1111 02:22:35.867860 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType I1111 02:22:35.867890 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun5 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0a32ef96-6d0c-415d-9fff-f5f376d27c96/globalmount with mount options([]) I1111 02:22:35.867904 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun5" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun5]) ... skipping 54 lines ... I1111 02:22:44.358492 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:22:44.358509 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4d5cd466-8cab-4c69-beee-d69091dce53a","csi.storage.k8s.io/pvc/name":"test.csi.azure.combqk8w","csi.storage.k8s.io/pvc/namespace":"multivolume-5806","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a"} I1111 02:22:44.358643 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4d5cd466-8cab-4c69-beee-d69091dce53a-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:22:47.616361 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:47.616448 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:47.616596 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:47.616739 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:22:47.616811 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:22:47.676029 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:22:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bbb01a84-6167-11ed-ae2b-6045bd88ebdb" I1111 02:22:47.681174 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds I1111 02:22:47.681728 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bbb01a84-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5729845 I1111 02:22:47.687101 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:47.805505 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:22:48.882491 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume ... skipping 15 lines ... W1111 02:22:49.002679 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0a32ef96-6d0c-415d-9fff-f5f376d27c96/globalmount" is not a mountpoint, deleting I1111 02:22:49.002769 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-0a32ef96-6d0c-415d-9fff-f5f376d27c96/globalmount successfully I1111 02:22:49.002789 1 utils.go:85] GRPC response: {} I1111 02:22:59.197695 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:22:59.197717 1 utils.go:79] GRPC request: {} I1111 02:22:59.197761 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:23:16.098127 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:23:16.098180 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b9b5ea77-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4d5cd466-8cab-4c69-beee-d69091dce53a" "latency"=31739488132 I1111 02:23:16.098205 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b9b5ea77-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31739574532 I1111 02:23:16.112044 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:23:16.112099 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b9b5c7f4-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9" "latency"=31754249348 I1111 02:23:16.112127 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b9b5c7f4-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31754378950 I1111 02:23:16.695833 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 02:23:16.695901 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:23:16.695944 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/globalmount with mount options([nouuid]) I1111 02:23:16.695971 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 211 lines ... I1111 02:23:35.890399 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/globalmount /var/lib/kubelet/pods/b5abdb58-4e93-42fe-b59f-edc67b96cab2/volumes/kubernetes.io~csi/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/mount) I1111 02:23:35.890622 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/globalmount /var/lib/kubelet/pods/b5abdb58-4e93-42fe-b59f-edc67b96cab2/volumes/kubernetes.io~csi/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/mount) I1111 02:23:35.891267 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/globalmount at /var/lib/kubelet/pods/b5abdb58-4e93-42fe-b59f-edc67b96cab2/volumes/kubernetes.io~csi/pvc-03509e51-f46f-4dc7-b1e7-06600d90c1d9/mount successfully I1111 02:23:35.891294 1 utils.go:85] GRPC response: {} I1111 02:23:35.891450 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/globalmount at /var/lib/kubelet/pods/b5abdb58-4e93-42fe-b59f-edc67b96cab2/volumes/kubernetes.io~csi/pvc-4d5cd466-8cab-4c69-beee-d69091dce53a/mount successfully I1111 02:23:35.891470 1 utils.go:85] GRPC response: {} I1111 02:23:36.476020 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:23:36.476081 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="cda59559-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-77bbd751-4bbe-4ed0-9bf3-4596c75901a5" "latency"=18669982492 I1111 02:23:36.476112 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cda59559-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18670082392 I1111 02:23:36.491758 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:23:36.491832 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="cd9653e7-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-41381644-1141-4bd5-8764-c59bc24c27f1" "latency"=18785696563 I1111 02:23:36.491877 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cd9653e7-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18785826563 I1111 02:23:37.064830 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:23:37.064850 1 utils.go:79] GRPC request: {} I1111 02:23:37.064890 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:23:37.065792 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats ... skipping 78 lines ... I1111 02:23:45.306261 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:23:45.307147 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:23:45.307167 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f81cf66-89b0-419b-b85d-493960e6afba/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2f81cf66-89b0-419b-b85d-493960e6afba","csi.storage.k8s.io/pvc/name":"test.csi.azure.comqd76d","csi.storage.k8s.io/pvc/namespace":"provisioning-9310","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-2f81cf66-89b0-419b-b85d-493960e6afba"} I1111 02:23:45.307361 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2f81cf66-89b0-419b-b85d-493960e6afba-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:23:47.618559 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:23:47.618576 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:23:47.618670 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:23:47.618717 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:23:47.676022 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:23:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="df735eea-6167-11ed-ae2b-6045bd88ebdb" I1111 02:23:47.682330 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:23:47.682568 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="df735eea-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6620868 I1111 02:23:47.688842 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:23:47.806147 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync ... skipping 75 lines ... I1111 02:23:53.879004 1 utils.go:85] GRPC response: {} I1111 02:23:57.597776 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 195 items received I1111 02:23:57.601152 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=11183&timeout=8m27s&timeoutSeconds=507&watch=true 200 OK in 3 milliseconds I1111 02:23:59.197039 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:23:59.197059 1 utils.go:79] GRPC request: {} I1111 02:23:59.197104 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:24:07.164877 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:24:07.164945 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="de09f30e-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2f81cf66-89b0-419b-b85d-493960e6afba" "latency"=21857505339 I1111 02:24:07.164973 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="de09f30e-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21857642741 I1111 02:24:07.557659 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:24:07.557682 1 utils.go:79] GRPC request: {} I1111 02:24:07.557727 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:24:07.560728 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 56 lines ... W1111 02:24:14.628877 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f81cf66-89b0-419b-b85d-493960e6afba/globalmount" is not a mountpoint, deleting I1111 02:24:14.628955 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2f81cf66-89b0-419b-b85d-493960e6afba/globalmount successfully I1111 02:24:14.628983 1 utils.go:85] GRPC response: {} I1111 02:24:17.619375 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:17.619585 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:17.619649 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:17.619836 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:24:17.675958 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:24:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f15500aa-6167-11ed-ae2b-6045bd88ebdb" I1111 02:24:17.684081 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds I1111 02:24:17.684330 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f15500aa-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8408067 I1111 02:24:17.689544 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:17.806922 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:29.198181 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:24:29.198201 1 utils.go:79] GRPC request: {} I1111 02:24:29.198247 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:24:47.620093 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:47.620187 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:47.620197 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:47.620281 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:24:47.675530 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:24:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0336928c-6168-11ed-ae2b-6045bd88ebdb" I1111 02:24:47.682057 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:24:47.682338 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0336928c-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6833360 I1111 02:24:47.690547 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:47.807946 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:24:48.672071 1 reflector.go:559] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 127 items received I1111 02:24:48.673983 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=11626&timeout=7m43s&timeoutSeconds=463&watch=true 200 OK in 1 milliseconds I1111 02:24:59.197932 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:24:59.197954 1 utils.go:79] GRPC request: {} I1111 02:24:59.197998 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:25:08.433254 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:25:08.433322 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="eb4dd336-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a72d9a1e-5acf-466a-98ee-4d8302dca0fe" "latency"=60870683747 I1111 02:25:08.433360 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="eb4dd336-6167-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60870817948 I1111 02:25:10.276048 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdc under /dev/disk/azure/scsi1/ I1111 02:25:10.276098 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun5. perfProfile none accountType I1111 02:25:10.276134 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun5 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a72d9a1e-5acf-466a-98ee-4d8302dca0fe/globalmount with mount options([]) I1111 02:25:10.276153 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun5" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun5]) ... skipping 91 lines ... I1111 02:25:29.197773 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:25:29.197792 1 utils.go:79] GRPC request: {} I1111 02:25:29.197834 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:25:47.621625 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:47.621649 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:47.621767 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:47.621795 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:25:47.621818 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:25:47.621848 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:25:47.621863 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:25:47.676116 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:25:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="26f9ef70-6168-11ed-ae2b-6045bd88ebdb" I1111 02:25:47.682279 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:25:47.683408 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="26f9ef70-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7343864 I1111 02:25:47.691654 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:47.809076 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:59.197661 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:25:59.197691 1 utils.go:79] GRPC request: {} I1111 02:25:59.197738 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:26:02.600491 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 367 items received I1111 02:26:02.607035 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=12534&timeout=5m28s&timeoutSeconds=328&watch=true 200 OK in 6 milliseconds I1111 02:26:17.622698 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:17.622795 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:17.622826 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:17.622924 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:17.622956 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:17.622975 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:17.623008 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:17.676209 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:26:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="38db9657-6168-11ed-ae2b-6045bd88ebdb" I1111 02:26:17.683540 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds I1111 02:26:17.683797 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="38db9657-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7622563 I1111 02:26:17.692043 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:17.809477 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:29.197613 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:26:29.197638 1 utils.go:79] GRPC request: {} I1111 02:26:29.197685 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:26:47.623187 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:47.623177 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:47.623318 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:47.623401 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:47.623431 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:47.623480 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:47.623518 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:47.675601 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:26:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="4abd211c-6168-11ed-ae2b-6045bd88ebdb" I1111 02:26:47.682181 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:26:47.682976 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="4abd211c-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7430265 I1111 02:26:47.692221 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:47.810562 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:59.198477 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:26:59.198496 1 utils.go:79] GRPC request: {} I1111 02:26:59.198536 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:27:17.624308 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:17.624379 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:17.624386 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:17.624420 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:17.624435 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:17.624446 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:17.624456 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:17.675760 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:27:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5c9ecb2a-6168-11ed-ae2b-6045bd88ebdb" I1111 02:27:17.685721 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 9 milliseconds I1111 02:27:17.686620 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5c9ecb2a-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10894991 I1111 02:27:17.692786 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:17.811129 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:19.604400 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 55 items received I1111 02:27:19.607937 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=12978&timeout=7m58s&timeoutSeconds=478&watch=true 200 OK in 3 milliseconds E1111 02:27:27.967902 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1b3b10b1-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845" "latency"=119997513172 E1111 02:27:27.967947 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1b3b10b1-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119997712473 E1111 02:27:27.967962 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845) to be attached to node (k8s-agentpool1-30725374-vmss000001): context canceled E1111 02:27:28.167188 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1b596c4a-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-20e42213-59b8-41b3-85a4-683078bfbb02" "latency"=119997867735 E1111 02:27:28.167256 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1b596c4a-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119998060136 E1111 02:27:28.167280 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-20e42213-59b8-41b3-85a4-683078bfbb02) to be attached to node (k8s-agentpool1-30725374-vmss000001): context deadline exceeded E1111 02:27:28.368615 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1b7815af-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e496bae9-5f84-4a37-bc4d-5d411c91560f" "latency"=119998370603 E1111 02:27:28.368676 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1b7815af-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119998534704 E1111 02:27:28.368694 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e496bae9-5f84-4a37-bc4d-5d411c91560f) to be attached to node (k8s-agentpool1-30725374-vmss000001): context canceled I1111 02:27:28.551412 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:27:28.551433 1 utils.go:79] GRPC request: {} I1111 02:27:28.551473 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:27:28.552197 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:27:28.552219 1 utils.go:79] GRPC request: {} I1111 02:27:28.552251 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:27:28.553032 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:27:28.553052 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845","csi.storage.k8s.io/pvc/name":"inline-volume-tester-w9jm8-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-6041","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845"} I1111 02:27:28.553242 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845-k8s-agentpool1-30725374-vmss000001-attachment) E1111 02:27:28.567365 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1b966d78-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063" "latency"=119998233261 E1111 02:27:28.567423 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1b966d78-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119998426763 E1111 02:27:28.567443 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063) to be attached to node (k8s-agentpool1-30725374-vmss000001): context deadline exceeded I1111 02:27:28.752926 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:27:28.752949 1 utils.go:79] GRPC request: {} I1111 02:27:28.752982 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:27:28.753757 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:27:28.753774 1 utils.go:79] GRPC request: {} I1111 02:27:28.753803 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 21 lines ... I1111 02:27:29.197613 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:27:29.197633 1 utils.go:79] GRPC request: {} I1111 02:27:29.197670 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:27:47.624433 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:47.624499 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:47.624558 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:47.624604 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:47.624621 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:47.624653 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:47.624669 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:47.676110 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:27:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="6e807a87-6168-11ed-ae2b-6045bd88ebdb" I1111 02:27:47.682547 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:27:47.682773 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="6e807a87-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6725757 I1111 02:27:47.693243 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:47.811563 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:59.198165 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:27:59.198190 1 utils.go:79] GRPC request: {} I1111 02:27:59.198231 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:28:17.624540 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:17.624664 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:17.624712 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:28:17.624733 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:28:17.624753 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:28:17.624755 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:17.624770 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:28:17.676089 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:28:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="80621d78-6168-11ed-ae2b-6045bd88ebdb" I1111 02:28:17.681761 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:28:17.681971 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="80621d78-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5928848 I1111 02:28:17.693626 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:17.812042 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:29.198007 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:28:29.198035 1 utils.go:79] GRPC request: {} I1111 02:28:29.198083 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:28:35.367448 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:28:35.367531 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="637737be-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063" "latency"=66207023744 I1111 02:28:35.367564 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="637737be-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66207160045 I1111 02:28:35.373227 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:28:35.373276 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="631a91da-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845" "latency"=66819974678 I1111 02:28:35.373298 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="631a91da-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66820074579 I1111 02:28:35.374991 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:28:35.375031 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="635861a7-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e496bae9-5f84-4a37-bc4d-5d411c91560f" "latency"=66416647834 I1111 02:28:35.375048 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="635861a7-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66416739135 I1111 02:28:35.375441 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:28:35.375492 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="63395269-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-20e42213-59b8-41b3-85a4-683078bfbb02" "latency"=66620659576 I1111 02:28:35.375511 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="63395269-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=66620752577 I1111 02:28:36.313150 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I1111 02:28:36.313207 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:28:36.313221 1 utils.go:85] GRPC response: {} I1111 02:28:36.313238 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/ ... skipping 74 lines ... I1111 02:28:38.652800 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:28:38.653480 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:28:38.653499 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"4"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261","csi.storage.k8s.io/pvc/name":"test.csi.azure.com72qjx","csi.storage.k8s.io/pvc/namespace":"multivolume-7189","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261"} I1111 02:28:38.653654 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:28:47.625506 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:47.625555 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:47.625600 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:28:47.625641 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:28:47.625659 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:47.675901 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:28:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9243b9bc-6168-11ed-ae2b-6045bd88ebdb" I1111 02:28:47.694253 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:47.696990 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 20 milliseconds I1111 02:28:47.698626 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9243b9bc-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=22737601 I1111 02:28:47.812999 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync ... skipping 28 lines ... I1111 02:28:50.274481 1 utils.go:85] GRPC response: {} I1111 02:28:50.274445 1 utils.go:79] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-20e42213-59b8-41b3-85a4-683078bfbb02","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-20e42213-59b8-41b3-85a4-683078bfbb02"} I1111 02:28:50.274529 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-20e42213-59b8-41b3-85a4-683078bfbb02 W1111 02:28:50.274781 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-20e42213-59b8-41b3-85a4-683078bfbb02" is not a mountpoint, deleting I1111 02:28:50.274825 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-20e42213-59b8-41b3-85a4-683078bfbb02 successfully I1111 02:28:50.274831 1 utils.go:85] GRPC response: {} I1111 02:28:50.957229 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:28:50.957291 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8cb4b6c5-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1466ecb7-5bce-4d42-9de7-fc40b1aaf05c" "latency"=12607198905 I1111 02:28:50.957326 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8cb4b6c5-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12607319205 I1111 02:28:50.959528 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:28:50.959605 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8ce30b46-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a85b59b6-6ba5-4d7a-bdef-ef29b3752261" "latency"=12305889018 I1111 02:28:50.959646 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8ce30b46-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12305996920 I1111 02:28:51.541216 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun5 by sdg under /dev/disk/azure/scsi1/ I1111 02:28:51.541216 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun4 by sdh under /dev/disk/azure/scsi1/ I1111 02:28:51.541301 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun4. perfProfile none accountType I1111 02:28:51.541321 1 utils.go:85] GRPC response: {} ... skipping 114 lines ... W1111 02:29:11.976814 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845" is not a mountpoint, deleting I1111 02:29:11.976873 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-ec52ac7a-d0cf-4239-a506-aeea4e409845 successfully I1111 02:29:11.976882 1 utils.go:85] GRPC response: {} W1111 02:29:11.976957 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063" is not a mountpoint, deleting I1111 02:29:11.977046 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-d13af50c-86a9-4b8c-81f7-7f377bda5063 successfully I1111 02:29:11.977066 1 utils.go:85] GRPC response: {} I1111 02:29:13.238482 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:29:13.238529 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="95cdd07d-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-09de6e98-ef97-4118-9359-0dcf58025169" "latency"=19624459441 I1111 02:29:13.238549 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="95cdd07d-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19624548341 I1111 02:29:13.239363 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:29:13.239413 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="95cd9d08-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-65b99319-536b-4adb-acd1-8410dc77d4c5" "latency"=19626631161 I1111 02:29:13.239438 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="95cd9d08-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19626756762 I1111 02:29:13.851839 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun7 by sdj under /dev/disk/azure/scsi1/ I1111 02:29:13.851840 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun6 by sdi under /dev/disk/azure/scsi1/ I1111 02:29:13.851899 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun7. perfProfile none accountType I1111 02:29:13.851931 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun6. perfProfile none accountType ... skipping 101 lines ... I1111 02:29:43.537312 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:29:43.537328 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"8"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-71fe6150-d0bf-4346-b3c2-a1365614f344/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-71fe6150-d0bf-4346-b3c2-a1365614f344","csi.storage.k8s.io/pvc/name":"inline-volume-tester-bxf4c-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-696","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-71fe6150-d0bf-4346-b3c2-a1365614f344"} I1111 02:29:43.537500 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-71fe6150-d0bf-4346-b3c2-a1365614f344-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:29:47.626745 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:29:47.626761 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:29:47.626771 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:29:47.626956 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:29:47.676250 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:29:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b6070ad5-6168-11ed-ae2b-6045bd88ebdb" I1111 02:29:47.682119 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:29:47.682338 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b6070ad5-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6187447 I1111 02:29:47.696305 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:29:47.814619 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:29:59.043105 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:29:59.043160 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b38f8871-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-71fe6150-d0bf-4346-b3c2-a1365614f344" "latency"=15505594536 I1111 02:29:59.043185 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b38f8871-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15505701738 I1111 02:29:59.197710 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:29:59.197733 1 utils.go:79] GRPC request: {} I1111 02:29:59.197780 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:29:59.580145 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 55 lines ... I1111 02:30:00.373465 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:30:00.373481 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"9"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-59e83969-58ae-46ee-b1d2-b773a4527216","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-59e83969-58ae-46ee-b1d2-b773a4527216","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzxmlz","csi.storage.k8s.io/pvc/namespace":"multivolume-2130","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-59e83969-58ae-46ee-b1d2-b773a4527216"} I1111 02:30:00.373640 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-59e83969-58ae-46ee-b1d2-b773a4527216-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:30:17.626860 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:17.626943 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:17.626956 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:17.627055 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:30:17.627082 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:30:17.627111 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:30:17.676460 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:30:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c7e8b745-6168-11ed-ae2b-6045bd88ebdb" I1111 02:30:17.681647 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds I1111 02:30:17.682065 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c7e8b745-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5635942 I1111 02:30:17.696914 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:17.815287 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:29.197767 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe ... skipping 18 lines ... W1111 02:30:35.189670 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-71fe6150-d0bf-4346-b3c2-a1365614f344/globalmount" is not a mountpoint, deleting I1111 02:30:35.189735 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-71fe6150-d0bf-4346-b3c2-a1365614f344/globalmount successfully I1111 02:30:35.189748 1 utils.go:85] GRPC response: {} I1111 02:30:47.627862 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:47.627977 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:47.628028 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:47.628209 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:30:47.628324 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:30:47.628364 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:30:47.675829 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:30:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d9ca425c-6168-11ed-ae2b-6045bd88ebdb" I1111 02:30:47.682589 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:30:47.682808 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d9ca425c-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7025357 I1111 02:30:47.697971 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:47.815701 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:30:50.026098 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:30:50.026180 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bd1fbd26-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-20e42213-59b8-41b3-85a4-683078bfbb02" "latency"=50444039056 I1111 02:30:50.026219 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bd1fbd26-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=50444168858 I1111 02:30:50.030993 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:30:50.031036 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bd988621-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-59e83969-58ae-46ee-b1d2-b773a4527216" "latency"=49657349381 I1111 02:30:50.031056 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bd988621-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=49657435482 I1111 02:30:50.031201 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:30:50.031252 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bd2c3c15-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e496bae9-5f84-4a37-bc4d-5d411c91560f" "latency"=50367226629 I1111 02:30:50.031284 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bd2c3c15-6168-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=50367343731 I1111 02:30:50.644325 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdf under /dev/disk/azure/scsi1/ I1111 02:30:50.644375 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:30:50.644387 1 utils.go:85] GRPC response: {} I1111 02:30:50.644972 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun9 by sde under /dev/disk/azure/scsi1/ ... skipping 165 lines ... I1111 02:33:03.488882 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:33:03.488899 1 utils.go:79] GRPC request: {} I1111 02:33:03.488937 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:33:03.489703 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:33:03.489717 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-59e83969-58ae-46ee-b1d2-b773a4527216","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-59e83969-58ae-46ee-b1d2-b773a4527216","csi.storage.k8s.io/pvc/name":"test.csi.azure.comzxmlz","csi.storage.k8s.io/pvc/namespace":"multivolume-2130","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-59e83969-58ae-46ee-b1d2-b773a4527216"} I1111 02:33:03.489901 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-59e83969-58ae-46ee-b1d2-b773a4527216-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:33:12.166239 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:33:12.166302 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2a333978-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-6f13450e-7108-4e37-91bc-2507a8873ea1" "latency"=9584815902 I1111 02:33:12.166330 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2a333978-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9584928204 I1111 02:33:12.166370 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:33:12.166417 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2abdd922-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-59e83969-58ae-46ee-b1d2-b773a4527216" "latency"=8676467213 I1111 02:33:12.166435 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2abdd922-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8676554415 I1111 02:33:12.759431 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/ I1111 02:33:12.759475 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:33:12.759486 1 utils.go:85] GRPC response: {} I1111 02:33:12.759805 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ ... skipping 80 lines ... I1111 02:33:29.197889 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:33:29.197916 1 utils.go:79] GRPC request: {} I1111 02:33:29.197970 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:33:47.631134 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:33:47.631162 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:33:47.631258 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:33:47.631383 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:33:47.675579 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:33:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="45140a59-6169-11ed-ae2b-6045bd88ebdb" I1111 02:33:47.680727 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds I1111 02:33:47.681461 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="45140a59-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5932447 I1111 02:33:47.700905 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:33:47.819161 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:33:48.991335 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 2 lines ... I1111 02:33:48.992238 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I1111 02:33:48.992252 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-59e83969-58ae-46ee-b1d2-b773a4527216","volume_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-59e83969-58ae-46ee-b1d2-b773a4527216/dev/000ca66c-7dd5-443f-93e5-13ea038fc628"} I1111 02:33:48.992951 1 utils.go:85] GRPC response: {"usage":[{"total":5368709120,"unit":1}]} I1111 02:33:59.197080 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:33:59.197099 1 utils.go:79] GRPC request: {} I1111 02:33:59.197141 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:34:09.635014 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:34:09.635073 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="397528c4-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154" "latency"=41455642678 I1111 02:34:09.635096 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="397528c4-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=41455763478 I1111 02:34:10.223876 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/ I1111 02:34:10.223923 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I1111 02:34:10.223936 1 utils.go:85] GRPC response: {} I1111 02:34:10.239022 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 33 lines ... I1111 02:34:11.571268 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:34:11.571287 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a67a0ecf-f127-4252-9abc-8f1327f68862/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a67a0ecf-f127-4252-9abc-8f1327f68862","csi.storage.k8s.io/pvc/name":"pvc-5g7jp","csi.storage.k8s.io/pvc/namespace":"provisioning-3371","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-a67a0ecf-f127-4252-9abc-8f1327f68862"} I1111 02:34:11.571461 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a67a0ecf-f127-4252-9abc-8f1327f68862-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:34:17.631881 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:34:17.631893 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:34:17.631936 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:34:17.632026 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:34:17.632064 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:34:17.640274 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume I1111 02:34:17.640291 1 utils.go:79] GRPC request: {"target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154"} I1111 02:34:17.640342 1 nodeserver_v2.go:369] NodeUnpublishVolume: unmounting volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154 on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f I1111 02:34:17.640367 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f" is a mountpoint, unmounting I1111 02:34:17.640380 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f W1111 02:34:17.642064 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-b7bf8fb7-deb0-4e89-adda-1c3e2c00f154/98ba362c-4b5e-4135-b34a-974e5e16c80f" is not a mountpoint, deleting ... skipping 30 lines ... W1111 02:34:20.928631 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-59e83969-58ae-46ee-b1d2-b773a4527216" is not a mountpoint, deleting I1111 02:34:20.928692 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-59e83969-58ae-46ee-b1d2-b773a4527216 successfully I1111 02:34:20.928709 1 utils.go:85] GRPC response: {} I1111 02:34:29.197465 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:34:29.197488 1 utils.go:79] GRPC request: {} I1111 02:34:29.197534 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:34:30.281679 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:34:30.281749 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="535244c0-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a67a0ecf-f127-4252-9abc-8f1327f68862" "latency"=18710229470 I1111 02:34:30.281784 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="535244c0-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18710342471 I1111 02:34:30.285101 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:34:30.285157 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="53523a06-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c968c265-3bba-493f-bae7-b67ba22bcf87" "latency"=18713889699 I1111 02:34:30.285178 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="53523a06-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=18714019500 I1111 02:34:30.868542 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I1111 02:34:30.868581 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:34:30.868612 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c968c265-3bba-493f-bae7-b67ba22bcf87/globalmount with mount options([]) I1111 02:34:30.868622 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 142 lines ... I1111 02:35:15.744602 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:35:17.607950 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 70 items received I1111 02:35:17.610401 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=17121&timeout=6m5s&timeoutSeconds=365&watch=true 200 OK in 2 milliseconds I1111 02:35:17.633650 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:17.633676 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:17.633745 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:17.633781 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:35:17.633827 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:35:17.676231 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:35:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7ab90d96-6169-11ed-ae2b-6045bd88ebdb" I1111 02:35:17.681561 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:35:17.682356 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7ab90d96-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6156049 I1111 02:35:17.703809 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:17.821310 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:29.198078 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:35:29.198103 1 utils.go:79] GRPC request: {} I1111 02:35:29.198146 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:35:38.686295 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:35:38.686359 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7992507a-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c" "latency"=22941705024 I1111 02:35:38.686384 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7992507a-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=22941795624 I1111 02:35:38.687348 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:35:38.687410 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7945787f-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6" "latency"=23446299025 I1111 02:35:38.687452 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7945787f-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=23446461726 I1111 02:35:39.273167 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdd under /dev/disk/azure/scsi1/ I1111 02:35:39.273207 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I1111 02:35:39.273236 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c/globalmount with mount options([]) I1111 02:35:39.273246 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3]) ... skipping 66 lines ... I1111 02:35:39.725482 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:35:39.726151 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:35:39.726166 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a","csi.storage.k8s.io/pvc/name":"test.csi.azure.comwmppf","csi.storage.k8s.io/pvc/namespace":"snapshotting-4341","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a"} I1111 02:35:39.726318 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:35:47.634500 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:47.634577 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:47.634589 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:35:47.634602 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:47.676038 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:35:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8c9aa8cf-6169-11ed-ae2b-6045bd88ebdb" I1111 02:35:47.682123 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:35:47.682576 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8c9aa8cf-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6577732 I1111 02:35:47.703875 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:47.822339 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:35:48.937606 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:35:48.937710 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="87dda225-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a" "latency"=9211323755 I1111 02:35:48.937732 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="87dda225-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9211434455 I1111 02:35:49.534289 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sde under /dev/disk/azure/scsi1/ I1111 02:35:49.534327 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:35:49.534356 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a/globalmount with mount options([]) I1111 02:35:49.534364 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 93 lines ... I1111 02:36:01.779940 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:36:01.779959 1 utils.go:79] GRPC request: {} I1111 02:36:01.779997 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:36:01.780702 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:36:01.780717 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94","csi.storage.k8s.io/pvc/name":"pvc-mjhpr","csi.storage.k8s.io/pvc/namespace":"snapshotting-4341","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94"} I1111 02:36:01.780889 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:36:13.844527 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:36:13.844600 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9502e4e3-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94" "latency"=12063638934 I1111 02:36:13.844634 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9502e4e3-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12063756135 I1111 02:36:14.355472 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:36:14.355500 1 utils.go:79] GRPC request: {} I1111 02:36:14.355545 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:36:14.358858 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 36 lines ... I1111 02:36:14.565177 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/globalmount /var/lib/kubelet/pods/72c9660b-69ed-458e-98bf-ec8c51fa7209/volumes/kubernetes.io~csi/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/mount) I1111 02:36:14.565943 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/globalmount at /var/lib/kubelet/pods/72c9660b-69ed-458e-98bf-ec8c51fa7209/volumes/kubernetes.io~csi/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/mount successfully I1111 02:36:14.565963 1 utils.go:85] GRPC response: {} I1111 02:36:17.635055 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:36:17.635114 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:36:17.635191 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:36:17.635341 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:36:17.675811 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:36:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9e7c4392-6169-11ed-ae2b-6045bd88ebdb" I1111 02:36:17.681330 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:36:17.681498 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9e7c4392-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5714245 I1111 02:36:17.704821 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:36:17.823154 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:36:29.197712 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe ... skipping 2 lines ... I1111 02:36:37.073249 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:36:37.073276 1 utils.go:79] GRPC request: {} I1111 02:36:37.073324 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:36:37.074061 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I1111 02:36:37.074082 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94","volume_path":"/var/lib/kubelet/pods/72c9660b-69ed-458e-98bf-ec8c51fa7209/volumes/kubernetes.io~csi/pvc-9d95475f-2338-495b-ae0f-7f84ecdf6b94/mount"} I1111 02:36:37.074180 1 utils.go:85] GRPC response: {"usage":[{"available":5179576320,"total":5196382208,"unit":1,"used":28672},{"available":327668,"total":327680,"unit":2,"used":12}]} I1111 02:36:44.583573 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:36:44.583647 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="9c8268e4-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-34903c00-ca40-488f-a7e7-3fe1806b3674" "latency"=30222956335 I1111 02:36:44.583683 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9c8268e4-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30223066137 I1111 02:36:45.172999 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/ I1111 02:36:45.173090 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:36:45.173131 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-34903c00-ca40-488f-a7e7-3fe1806b3674/globalmount with mount options([]) I1111 02:36:45.173153 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 169 lines ... I1111 02:37:23.459935 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:37:24.613122 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 220 items received I1111 02:37:24.614762 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=18592&timeout=7m17s&timeoutSeconds=437&watch=true 200 OK in 1 milliseconds I1111 02:37:29.197522 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:37:29.197545 1 utils.go:79] GRPC request: {} I1111 02:37:29.197589 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:37:45.314216 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:37:45.314287 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c5b21633-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a51dc9b0-7b91-42ed-abe4-206458934d79" "latency"=21854527623 I1111 02:37:45.314317 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c5b21633-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21854666724 I1111 02:37:45.315924 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:37:45.315991 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="c5b22096-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847" "latency"=21855993035 I1111 02:37:45.316021 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="c5b22096-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=21856097836 I1111 02:37:45.824206 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:37:45.824231 1 utils.go:79] GRPC request: {} I1111 02:37:45.824275 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:37:45.827291 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 69 lines ... I1111 02:37:46.414070 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/globalmount /var/lib/kubelet/pods/a743d533-5855-4339-af5a-0d139613a95c/volumes/kubernetes.io~csi/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/mount) I1111 02:37:46.414889 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/globalmount at /var/lib/kubelet/pods/a743d533-5855-4339-af5a-0d139613a95c/volumes/kubernetes.io~csi/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/mount successfully I1111 02:37:46.414903 1 utils.go:85] GRPC response: {} I1111 02:37:47.637308 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:37:47.637353 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:37:47.637315 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:37:47.637437 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:37:47.675595 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:37:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d421229a-6169-11ed-ae2b-6045bd88ebdb" I1111 02:37:47.681500 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:37:47.682451 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d421229a-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6908857 I1111 02:37:47.706612 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:37:47.824866 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:37:59.197472 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe ... skipping 118 lines ... I1111 02:38:04.733385 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847/globalmount /var/lib/kubelet/pods/dd3b6b97-012e-47ef-a2a8-33a6a1d70be9/volumes/kubernetes.io~csi/pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847/mount) I1111 02:38:04.733662 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/globalmount /var/lib/kubelet/pods/dd3b6b97-012e-47ef-a2a8-33a6a1d70be9/volumes/kubernetes.io~csi/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/mount) I1111 02:38:04.734106 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847/globalmount at /var/lib/kubelet/pods/dd3b6b97-012e-47ef-a2a8-33a6a1d70be9/volumes/kubernetes.io~csi/pvc-00efa424-3c77-4a8e-9dc1-a1e7b3b1f847/mount successfully I1111 02:38:04.734116 1 utils.go:85] GRPC response: {} I1111 02:38:04.734578 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/globalmount at /var/lib/kubelet/pods/dd3b6b97-012e-47ef-a2a8-33a6a1d70be9/volumes/kubernetes.io~csi/pvc-a51dc9b0-7b91-42ed-abe4-206458934d79/mount successfully I1111 02:38:04.734594 1 utils.go:85] GRPC response: {} I1111 02:38:05.674369 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:38:05.674443 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d3075d90-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-50c4b81c-d020-4dad-a795-c865b1da561d" "latency"=19845441292 I1111 02:38:05.674469 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d3075d90-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19845551493 I1111 02:38:06.272374 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I1111 02:38:06.272420 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:38:06.272454 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-50c4b81c-d020-4dad-a795-c865b1da561d/globalmount with mount options([]) I1111 02:38:06.272467 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 104 lines ... I1111 02:38:29.197805 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:38:29.197824 1 utils.go:79] GRPC request: {} I1111 02:38:29.197861 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:38:47.638803 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:38:47.638851 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:38:47.638830 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:38:47.638999 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:38:47.676473 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:38:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f7e48aa3-6169-11ed-ae2b-6045bd88ebdb" I1111 02:38:47.682424 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:38:47.682641 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f7e48aa3-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6270047 I1111 02:38:47.708407 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:38:47.826829 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:38:59.197790 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:38:59.197809 1 utils.go:79] GRPC request: {} I1111 02:38:59.197857 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:39:03.797139 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 29 items received I1111 02:39:03.800459 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000001&resourceVersion=19263&timeout=7m44s&timeoutSeconds=464&watch=true 200 OK in 3 milliseconds I1111 02:39:06.585081 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:39:06.585146 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="eb197197-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-41e04970-9406-4ef2-8242-676cbc074a5d" "latency"=40372320520 I1111 02:39:06.585179 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="eb197197-6169-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=40372462220 I1111 02:39:07.216035 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:39:07.216059 1 utils.go:79] GRPC request: {} I1111 02:39:07.216098 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:39:07.219424 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 53 lines ... I1111 02:39:15.570173 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount I1111 02:39:15.570214 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount" is a mountpoint, unmounting I1111 02:39:15.570229 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount W1111 02:39:15.592605 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount" is not a mountpoint, deleting I1111 02:39:15.592698 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41e04970-9406-4ef2-8242-676cbc074a5d/globalmount successfully I1111 02:39:15.592720 1 utils.go:85] GRPC response: {} I1111 02:39:16.891232 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:39:16.891293 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="038ad6e0-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-938bb352-9562-491f-b7d6-05b0976e093b" "latency"=9670033046 I1111 02:39:16.891323 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="038ad6e0-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9670137546 I1111 02:39:17.486641 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I1111 02:39:17.486693 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:39:17.486733 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-938bb352-9562-491f-b7d6-05b0976e093b/globalmount with mount options([]) I1111 02:39:17.486751 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 68 lines ... I1111 02:39:29.197233 1 utils.go:79] GRPC request: {} I1111 02:39:29.197281 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:39:45.686082 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: Watch close - *v1.CustomResourceDefinition total 7 items received I1111 02:39:45.688535 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=16328&timeout=7m3s&timeoutSeconds=423&watch=true 200 OK in 2 milliseconds I1111 02:39:47.640001 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:47.640082 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:47.640115 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:39:47.640159 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:47.676460 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:39:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1ba7d192-616a-11ed-ae2b-6045bd88ebdb" I1111 02:39:47.681858 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:39:47.682400 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1ba7d192-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5973306 I1111 02:39:47.709575 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:47.828945 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:59.197924 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:39:59.197947 1 utils.go:79] GRPC request: {} I1111 02:39:59.197997 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:40:17.640474 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:40:17.640497 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:40:17.640541 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:40:17.640626 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:40:17.676023 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:40:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2d8962c8-616a-11ed-ae2b-6045bd88ebdb" I1111 02:40:17.681968 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:40:17.682398 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="2d8962c8-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6430456 I1111 02:40:17.710675 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:40:17.830015 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:40:20.303453 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:40:20.303521 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="0f03fd1a-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c790a21e-6466-4989-a38c-3c9f1bb85b5e" "latency"=53833342957 I1111 02:40:20.303545 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="0f03fd1a-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=53833462158 I1111 02:40:22.161787 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I1111 02:40:22.161835 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:40:22.161877 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c790a21e-6466-4989-a38c-3c9f1bb85b5e/globalmount with mount options([]) I1111 02:40:22.161892 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 60 lines ... I1111 02:40:37.480437 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:40:37.480452 1 utils.go:79] GRPC request: {} I1111 02:40:37.480480 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:40:37.481273 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:40:37.481293 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1","csi.storage.k8s.io/pvc/name":"test.csi.azure.com4h6ww","csi.storage.k8s.io/pvc/namespace":"multivolume-8476","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1"} I1111 02:40:37.481461 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:40:46.708563 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:40:46.708638 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3957766e-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1" "latency"=9227095568 I1111 02:40:46.708679 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3957766e-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9227225470 I1111 02:40:46.951275 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:40:46.951296 1 utils.go:79] GRPC request: {} I1111 02:40:46.951337 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:40:46.952077 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 8 lines ... I1111 02:40:47.300295 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) I1111 02:40:47.309763 1 mount_linux.go:490] Output: "" I1111 02:40:47.309787 1 mount_linux.go:449] Disk "/dev/disk/azure/scsi1/lun0" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/disk/azure/scsi1/lun0] I1111 02:40:47.641154 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:40:47.641199 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:40:47.641218 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:40:47.641291 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:40:47.675650 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:40:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3f6af5ff-616a-11ed-ae2b-6045bd88ebdb" I1111 02:40:47.681574 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:40:47.681903 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3f6af5ff-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6339366 I1111 02:40:47.711090 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:40:47.789098 1 mount_linux.go:459] Disk successfully formatted (mkfs): ext4 - /dev/disk/azure/scsi1/lun0 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount I1111 02:40:47.789131 1 mount_linux.go:477] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount ... skipping 86 lines ... I1111 02:41:09.879781 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount /var/lib/kubelet/pods/02ecbfaf-5b90-41b3-8999-13e662be968e/volumes/kubernetes.io~csi/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/mount) I1111 02:41:09.880639 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/globalmount at /var/lib/kubelet/pods/02ecbfaf-5b90-41b3-8999-13e662be968e/volumes/kubernetes.io~csi/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/mount successfully I1111 02:41:09.880661 1 utils.go:85] GRPC response: {} I1111 02:41:17.641809 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:17.641969 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:17.641997 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:17.642178 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:41:17.676462 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:41:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="514cba10-616a-11ed-ae2b-6045bd88ebdb" I1111 02:41:17.693381 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 16 milliseconds I1111 02:41:17.695089 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="514cba10-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18673684 I1111 02:41:17.711524 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:17.831884 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:22.611394 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 58 items received ... skipping 4 lines ... I1111 02:41:37.075269 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:41:37.075290 1 utils.go:79] GRPC request: {} I1111 02:41:37.075325 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:41:37.076096 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats I1111 02:41:37.076116 1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1","volume_path":"/var/lib/kubelet/pods/02ecbfaf-5b90-41b3-8999-13e662be968e/volumes/kubernetes.io~csi/pvc-127c1e01-98b3-49dc-91fb-bd0aaec1ead1/mount"} I1111 02:41:37.076192 1 utils.go:85] GRPC response: {"usage":[{"available":5179576320,"total":5196382208,"unit":1,"used":28672},{"available":327668,"total":327680,"unit":2,"used":12}]} I1111 02:41:37.568416 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:41:37.568489 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3efce76b-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f7af547e-1ffd-4b0e-8a57-45ce93934d64" "latency"=50614128239 I1111 02:41:37.568519 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3efce76b-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=50614226342 I1111 02:41:38.160858 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdc under /dev/disk/azure/scsi1/ I1111 02:41:38.160908 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:41:38.160920 1 utils.go:85] GRPC response: {} I1111 02:41:38.164623 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 38 lines ... W1111 02:41:46.986634 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f7af547e-1ffd-4b0e-8a57-45ce93934d64" is not a mountpoint, deleting I1111 02:41:46.986696 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-f7af547e-1ffd-4b0e-8a57-45ce93934d64 successfully I1111 02:41:46.986704 1 utils.go:85] GRPC response: {} I1111 02:41:47.642487 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:47.642516 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:47.642506 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:47.642618 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:41:47.675938 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:41:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="632e4849-616a-11ed-ae2b-6045bd88ebdb" I1111 02:41:47.681431 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:41:47.681666 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="632e4849-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5788962 I1111 02:41:47.711786 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:47.832051 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:41:59.197845 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:41:59.197870 1 utils.go:79] GRPC request: {} I1111 02:41:59.197917 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:42:13.103262 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:42:13.103337 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5e074f14-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d4853aa0-c900-469a-b015-4429594b1f96" "latency"=34071376356 I1111 02:42:13.103370 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5e074f14-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34071515857 I1111 02:42:13.696273 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sde under /dev/disk/azure/scsi1/ I1111 02:42:13.696325 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:42:13.696352 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d4853aa0-c900-469a-b015-4429594b1f96/globalmount with mount options([]) I1111 02:42:13.696361 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 83 lines ... I1111 02:42:30.093856 1 utils.go:79] GRPC request: {} I1111 02:42:30.093888 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:42:30.094520 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:42:30.094534 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"3"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e","csi.storage.k8s.io/pvc/name":"pvc-2bwlb","csi.storage.k8s.io/pvc/namespace":"snapshotting-955","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e"} I1111 02:42:30.094672 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:42:47.643677 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:42:47.643783 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:42:47.643806 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:42:47.643831 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:42:47.676189 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:42:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="86f198fd-616a-11ed-ae2b-6045bd88ebdb" I1111 02:42:47.681114 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds I1111 02:42:47.681337 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="86f198fd-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5182946 I1111 02:42:47.713116 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:42:47.833519 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:42:59.198104 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:42:59.198132 1 utils.go:79] GRPC request: {} I1111 02:42:59.198178 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:43:17.644506 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:17.644583 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:17.644581 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:17.644706 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:43:17.676042 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:43:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="98d335a5-616a-11ed-ae2b-6045bd88ebdb" I1111 02:43:17.681984 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:43:17.682539 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="98d335a5-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6547355 I1111 02:43:17.713988 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:17.834385 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:29.197262 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:43:29.197286 1 utils.go:79] GRPC request: {} I1111 02:43:29.197336 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:43:39.270059 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:43:39.270114 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7c76dff2-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e" "latency"=69175367359 I1111 02:43:39.270140 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7c76dff2-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=69175484960 I1111 02:43:39.863159 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/ I1111 02:43:39.863211 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I1111 02:43:39.863246 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount with mount options([]) I1111 02:43:39.863258 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3]) ... skipping 54 lines ... I1111 02:43:41.532248 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:43:41.532263 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"5"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c","csi.storage.k8s.io/pvc/name":"pvc-tfmgp","csi.storage.k8s.io/pvc/namespace":"provisioning-6901","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c"} I1111 02:43:41.532421 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:43:47.644918 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:47.645061 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:47.645095 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:47.645279 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:43:47.645355 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:43:47.645429 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:43:47.676518 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:43:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="aab4ec19-616a-11ed-ae2b-6045bd88ebdb" I1111 02:43:47.683633 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:43:47.683903 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="aab4ec19-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7400278 I1111 02:43:47.714459 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:47.834876 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:59.198047 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe ... skipping 23 lines ... I1111 02:44:15.983455 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount W1111 02:44:15.999503 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount" is not a mountpoint, deleting I1111 02:44:15.999556 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fb156490-dda7-4713-ac6e-5c41b1cd6a7e/globalmount successfully I1111 02:44:15.999565 1 utils.go:85] GRPC response: {} I1111 02:44:17.645759 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:44:17.645848 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:44:17.645869 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:44:17.645879 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:44:17.645903 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:44:17.645918 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:44:17.676323 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:44:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bc968785-616a-11ed-ae2b-6045bd88ebdb" I1111 02:44:17.683160 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:44:17.683329 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="bc968785-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7024458 I1111 02:44:17.714662 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:44:17.835046 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:44:29.197881 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:44:29.197903 1 utils.go:79] GRPC request: {} I1111 02:44:29.197950 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:44:41.024011 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:44:41.024068 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a70b690b-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cacecee8-1f4b-4286-a42c-6c1dc2988e3c" "latency"=59491589335 I1111 02:44:41.024095 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a70b690b-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=59491691435 I1111 02:44:41.028282 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:44:41.028326 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a6be1b8e-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d021c5d3-9653-4df4-b6f0-2b4b051c35d0" "latency"=60002448197 I1111 02:44:41.028346 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a6be1b8e-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60002555798 I1111 02:44:41.029088 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:44:41.029161 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a6be1cc1-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-78e6944f-19bb-4f8e-81ef-8dbae9ae0778" "latency"=60003269204 I1111 02:44:41.029196 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a6be1cc1-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=60003369004 I1111 02:44:41.615896 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 190 items received I1111 02:44:41.619564 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=22232&timeout=6m23s&timeoutSeconds=383&watch=true 200 OK in 3 milliseconds I1111 02:44:41.980003 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdf under /dev/disk/azure/scsi1/ I1111 02:44:41.980072 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType ... skipping 146 lines ... I1111 02:45:29.197154 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:45:29.197178 1 utils.go:79] GRPC request: {} I1111 02:45:29.197215 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:45:47.646772 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:47.646924 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:47.646965 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:47.647038 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:45:47.676340 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:45:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f23b70b5-616a-11ed-ae2b-6045bd88ebdb" I1111 02:45:47.681764 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:45:47.682012 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f23b70b5-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5709548 I1111 02:45:47.715813 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:47.836054 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:52.807778 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:45:52.807860 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="e4173e7a-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1216e252-79f7-49d1-8737-6722ac45923d" "latency"=28856781576 I1111 02:45:52.807902 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e4173e7a-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=28856921278 I1111 02:45:53.871356 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:45:53.871381 1 utils.go:79] GRPC request: {} I1111 02:45:53.871418 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:45:53.874540 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 41 lines ... I1111 02:45:59.197925 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:46:13.607155 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 118 items received I1111 02:46:13.609604 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=23172&timeout=7m31s&timeoutSeconds=451&watch=true 200 OK in 2 milliseconds I1111 02:46:17.647722 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:46:17.647778 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:46:17.647804 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:46:17.647903 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:46:17.676158 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:46:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="041d0b99-616b-11ed-ae2b-6045bd88ebdb" I1111 02:46:17.680774 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds I1111 02:46:17.681091 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="041d0b99-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=4995241 I1111 02:46:17.716211 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:46:17.836627 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:46:23.433289 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:46:23.433369 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f5ed7bc3-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1d184be2-3854-4618-a760-98f325e099c3" "latency"=29557007399 I1111 02:46:23.433411 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f5ed7bc3-616a-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=29557122600 I1111 02:46:24.028133 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 02:46:24.028191 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:46:24.028224 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1d184be2-3854-4618-a760-98f325e099c3/globalmount with mount options([]) I1111 02:46:24.028240 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 89 lines ... I1111 02:47:05.879394 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:47:05.879410 1 utils.go:79] GRPC request: {} I1111 02:47:05.879446 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:47:05.880257 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:47:05.880276 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa","csi.storage.k8s.io/pvc/name":"pvc-mclzr","csi.storage.k8s.io/pvc/namespace":"provisioning-200","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa"} I1111 02:47:05.880450 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:47:14.288439 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:47:14.288505 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="20d871ed-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-341c5962-95ed-4733-8b4a-433fbb9dd5aa" "latency"=8407969841 I1111 02:47:14.288529 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="20d871ed-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8408095842 I1111 02:47:14.911051 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 02:47:14.911100 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:47:14.911111 1 utils.go:85] GRPC response: {} I1111 02:47:14.916648 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 21 lines ... I1111 02:47:16.166352 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:47:16.166371 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f505dad6-786d-4e08-a7ce-a993beed5714","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-2bf67-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-5208","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-f505dad6-786d-4e08-a7ce-a993beed5714"} I1111 02:47:16.166558 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-f505dad6-786d-4e08-a7ce-a993beed5714-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:47:17.648341 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:17.648408 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:17.648421 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:17.648481 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:47:17.675868 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:47:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="27e0466e-616b-11ed-ae2b-6045bd88ebdb" I1111 02:47:17.681965 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:47:17.682205 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="27e0466e-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6398052 I1111 02:47:17.718262 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:17.837650 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:24.252667 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume ... skipping 16 lines ... I1111 02:47:29.197342 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:47:29.197362 1 utils.go:79] GRPC request: {} I1111 02:47:29.197401 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:47:47.648801 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:47.648870 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:47.648897 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:47.648944 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:47:47.676245 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:47:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="39c1f935-616b-11ed-ae2b-6045bd88ebdb" I1111 02:47:47.682660 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:47:47.682879 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="39c1f935-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6658355 I1111 02:47:47.718926 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:47.838301 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:47:50.356817 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:47:50.356890 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="26f9fad4-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f505dad6-786d-4e08-a7ce-a993beed5714" "latency"=34190270190 I1111 02:47:50.356941 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="26f9fad4-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34190379891 I1111 02:47:50.811862 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:47:50.811882 1 utils.go:79] GRPC request: {} I1111 02:47:50.811915 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:47:50.815153 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 49 lines ... I1111 02:47:52.855754 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/globalmount /var/lib/kubelet/pods/6f927703-35aa-46f6-97fd-c04319cb012e/volumes/kubernetes.io~csi/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/mount) I1111 02:47:52.856524 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/globalmount at /var/lib/kubelet/pods/6f927703-35aa-46f6-97fd-c04319cb012e/volumes/kubernetes.io~csi/pvc-f505dad6-786d-4e08-a7ce-a993beed5714/mount successfully I1111 02:47:52.856538 1 utils.go:85] GRPC response: {} I1111 02:47:59.198028 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:47:59.198054 1 utils.go:79] GRPC request: {} I1111 02:47:59.198100 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:48:05.817481 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:48:05.817536 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3ba12dc6-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ce77a2e2-f9fb-4c4f-b3d4-b38e7d635385" "latency"=15000724925 I1111 02:48:05.817563 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3ba12dc6-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15000836426 I1111 02:48:05.822252 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:48:05.822313 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="3bc04777-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f42a016a-7ee3-409b-ba03-5dc001702368" "latency"=14801703889 I1111 02:48:05.822349 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="3bc04777-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14801801789 I1111 02:48:06.234737 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:48:06.234763 1 utils.go:79] GRPC request: {} I1111 02:48:06.234800 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:48:06.235627 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 66 lines ... I1111 02:48:06.922137 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/globalmount /var/lib/kubelet/pods/4dee0fad-d284-4c05-93a1-b000ddac5a20/volumes/kubernetes.io~csi/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/mount) I1111 02:48:06.922892 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/globalmount at /var/lib/kubelet/pods/4dee0fad-d284-4c05-93a1-b000ddac5a20/volumes/kubernetes.io~csi/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/mount successfully I1111 02:48:06.922909 1 utils.go:85] GRPC response: {} I1111 02:48:17.649601 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:17.649655 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:17.649655 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:17.649786 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:48:17.676008 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:48:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="4ba39327-616b-11ed-ae2b-6045bd88ebdb" I1111 02:48:17.682573 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:48:17.682765 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="4ba39327-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6779956 I1111 02:48:17.720029 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:17.839365 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:19.649436 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume ... skipping 31 lines ... W1111 02:48:19.771899 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce77a2e2-f9fb-4c4f-b3d4-b38e7d635385/globalmount" is not a mountpoint, deleting I1111 02:48:19.771955 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce77a2e2-f9fb-4c4f-b3d4-b38e7d635385/globalmount successfully I1111 02:48:19.771965 1 utils.go:85] GRPC response: {} W1111 02:48:19.774155 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/globalmount" is not a mountpoint, deleting I1111 02:48:19.774226 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f42a016a-7ee3-409b-ba03-5dc001702368/globalmount successfully I1111 02:48:19.774245 1 utils.go:85] GRPC response: {} I1111 02:48:21.160281 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:48:21.160361 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="44d21287-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8a0ffac8-d2fc-4d10-acf5-5eb4cafd289d" "latency"=14923623833 I1111 02:48:21.160405 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="44d21287-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=14923746833 I1111 02:48:21.760677 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/ I1111 02:48:21.760725 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I1111 02:48:21.760771 1 utils.go:85] GRPC response: {} I1111 02:48:21.766358 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 90 lines ... I1111 02:50:01.294998 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:50:01.295013 1 utils.go:79] GRPC request: {} I1111 02:50:01.295045 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:50:01.295837 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:50:01.295852 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef","csi.storage.k8s.io/pvc/name":"inline-volume-tester-p586b-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-3195","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef"} I1111 02:50:01.296019 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:50:11.287653 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:50:11.287729 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="8966bc8d-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef" "latency"=9991620697 I1111 02:50:11.287754 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8966bc8d-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9991753799 I1111 02:50:12.474401 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:50:12.474421 1 utils.go:79] GRPC request: {} I1111 02:50:12.474461 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:50:12.480156 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 37 lines ... I1111 02:50:13.522996 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/globalmount /var/lib/kubelet/pods/608bbc53-8ef5-4e5b-a1d1-c7922e2afb5e/volumes/kubernetes.io~csi/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/mount) I1111 02:50:13.523764 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/globalmount at /var/lib/kubelet/pods/608bbc53-8ef5-4e5b-a1d1-c7922e2afb5e/volumes/kubernetes.io~csi/pvc-2b6e2c4b-7568-4f22-9a24-1353ebcee5ef/mount successfully I1111 02:50:13.523780 1 utils.go:85] GRPC response: {} I1111 02:50:17.651137 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:50:17.651238 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:50:17.651269 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:50:17.651353 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:50:17.675511 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:50:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="932a0b5b-616b-11ed-ae2b-6045bd88ebdb" I1111 02:50:17.683290 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds I1111 02:50:17.683520 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="932a0b5b-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8044764 I1111 02:50:17.723693 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:50:17.842117 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:50:29.198160 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:50:29.198184 1 utils.go:79] GRPC request: {} I1111 02:50:29.198226 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:50:31.720779 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:50:31.720841 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="90119477-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-98ed955e-36b4-479d-976c-421724b1c02d" "latency"=19238797558 I1111 02:50:31.720866 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="90119477-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19238896959 I1111 02:50:32.315522 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 02:50:32.315572 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:50:32.315613 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-98ed955e-36b4-479d-976c-421724b1c02d/globalmount with mount options([]) I1111 02:50:32.315634 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 163 lines ... I1111 02:51:14.918768 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:51:14.918785 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e1731f59-32e9-439e-adef-6837d3c212ca/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e1731f59-32e9-439e-adef-6837d3c212ca","csi.storage.k8s.io/pvc/name":"test.csi.azure.combjp2v","csi.storage.k8s.io/pvc/namespace":"provisioning-4376","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e1731f59-32e9-439e-adef-6837d3c212ca"} I1111 02:51:14.918942 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e1731f59-32e9-439e-adef-6837d3c212ca-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:51:17.652949 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:51:17.652986 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:51:17.653051 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:51:17.653102 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:51:17.676521 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:51:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b6ed7644-616b-11ed-ae2b-6045bd88ebdb" I1111 02:51:17.682545 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:51:17.683057 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b6ed7644-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6593638 I1111 02:51:17.725505 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:51:17.843835 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:51:29.198088 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:51:29.198109 1 utils.go:79] GRPC request: {} I1111 02:51:29.198152 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:51:34.010043 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:51:34.010105 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="b548b422-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e1731f59-32e9-439e-adef-6837d3c212ca" "latency"=19091092102 I1111 02:51:34.010128 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="b548b422-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19091202002 I1111 02:51:35.688508 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 02:51:35.688830 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:51:35.688882 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e1731f59-32e9-439e-adef-6837d3c212ca/globalmount with mount options([]) I1111 02:51:35.688897 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 66 lines ... I1111 02:52:03.442629 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:52:03.442646 1 utils.go:79] GRPC request: {} I1111 02:52:03.442681 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:52:03.443530 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:52:03.443546 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3e25c7e8-8885-4454-ac16-3b390acd3669","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-7fl89-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-57","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669"} I1111 02:52:03.443735 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3e25c7e8-8885-4454-ac16-3b390acd3669-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:52:14.831526 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:52:14.831597 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d234ffbb-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3e25c7e8-8885-4454-ac16-3b390acd3669" "latency"=11387803437 I1111 02:52:14.831631 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d234ffbb-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11387913837 I1111 02:52:15.341329 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:52:15.341352 1 utils.go:79] GRPC request: {} I1111 02:52:15.341393 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:52:15.345158 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 13 lines ... I1111 02:52:16.734579 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:52:16.735460 1 utils.go:78] GRPC call: /csi.v1.Node/NodePublishVolume I1111 02:52:16.735480 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669","target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3e25c7e8-8885-4454-ac16-3b390acd3669","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-7fl89-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-57","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669"} I1111 02:52:17.653385 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:52:17.653422 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:52:17.653405 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:52:17.653492 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:52:17.675917 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:52:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="dab0a701-616b-11ed-ae2b-6045bd88ebdb" I1111 02:52:17.683547 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 7 milliseconds I1111 02:52:17.683763 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="dab0a701-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7885962 I1111 02:52:17.726869 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:52:17.845119 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:52:18.538167 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I1111 02:52:18.538208 1 nodeserver_v2.go:330] NodePublishVolume [block]: found device path /dev/disk/azure/scsi1/lun1 with LUN 1 I1111 02:52:18.538235 1 nodeprovisioner.go:164] EnsureBlockTargetReady [block]: making target file /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018 I1111 02:52:18.538300 1 nodeserver_v2.go:348] NodePublishVolume: mounting /dev/disk/azure/scsi1/lun1 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018 I1111 02:52:18.538314 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /dev/disk/azure/scsi1/lun1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018) I1111 02:52:18.539317 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /dev/disk/azure/scsi1/lun1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018) I1111 02:52:18.540163 1 nodeserver_v2.go:353] NodePublishVolume: mount /dev/disk/azure/scsi1/lun1 at /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669/07458416-4d83-4cc7-84ab-bb4c024e4018 successfully I1111 02:52:18.540183 1 utils.go:85] GRPC response: {} I1111 02:52:25.076479 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:52:25.076577 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d94d415c-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fee094c3-c0b8-4fc3-92db-e1aa52d27d18" "latency"=9729752897 I1111 02:52:25.076628 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d94d415c-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9729889097 I1111 02:52:25.668584 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I1111 02:52:25.668630 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:52:25.668658 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-fee094c3-c0b8-4fc3-92db-e1aa52d27d18/globalmount with mount options([]) I1111 02:52:25.668669 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 65 lines ... I1111 02:52:43.838317 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:52:43.838953 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:52:43.838971 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719","csi.storage.k8s.io/pvc/name":"pvc-psmfp","csi.storage.k8s.io/pvc/namespace":"snapshotting-6604","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719"} I1111 02:52:43.839112 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:52:47.654612 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:52:47.654760 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:52:47.654863 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:52:47.654744 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:52:47.676203 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:52:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="ec92556d-616b-11ed-ae2b-6045bd88ebdb" I1111 02:52:47.682538 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 6 milliseconds I1111 02:52:47.682760 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="ec92556d-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6592747 I1111 02:52:47.727929 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:52:47.845263 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync ... skipping 14 lines ... W1111 02:52:53.405164 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669" is not a mountpoint, deleting I1111 02:52:53.405238 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-3e25c7e8-8885-4454-ac16-3b390acd3669 successfully I1111 02:52:53.405253 1 utils.go:85] GRPC response: {} I1111 02:52:59.197920 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:52:59.197945 1 utils.go:79] GRPC request: {} I1111 02:52:59.197982 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:53:00.598540 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:53:00.598612 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="ea48d83b-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719" "latency"=16759424433 I1111 02:53:00.598643 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="ea48d83b-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16759540935 I1111 02:53:01.189824 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:53:01.189847 1 utils.go:79] GRPC request: {} I1111 02:53:01.189891 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:53:01.191493 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ ... skipping 36 lines ... I1111 02:53:01.325347 1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/globalmount /var/lib/kubelet/pods/8a80a59a-598c-4fd2-9c73-3ce7038d93d1/volumes/kubernetes.io~csi/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/mount) I1111 02:53:01.326087 1 nodeserver_v2.go:353] NodePublishVolume: mount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/globalmount at /var/lib/kubelet/pods/8a80a59a-598c-4fd2-9c73-3ce7038d93d1/volumes/kubernetes.io~csi/pvc-4a9c5546-7899-41b6-9b34-f82d5d7d8719/mount successfully I1111 02:53:01.326100 1 utils.go:85] GRPC response: {} I1111 02:53:17.655489 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:53:17.655516 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:53:17.655574 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:53:17.655611 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:53:17.675895 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:53:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="fe73ec6a-616b-11ed-ae2b-6045bd88ebdb" I1111 02:53:17.681444 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:53:17.682326 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="fe73ec6a-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6465151 I1111 02:53:17.728701 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:53:17.846078 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:53:27.803274 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 21 items received I1111 02:53:27.806673 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000001&resourceVersion=26749&timeout=5m15s&timeoutSeconds=315&watch=true 200 OK in 3 milliseconds I1111 02:53:29.197573 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:53:29.197600 1 utils.go:79] GRPC request: {} I1111 02:53:29.197650 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:53:31.297788 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:53:31.297865 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="f4a131ee-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b78f5c34-fd0f-4e65-be44-ef642570638f" "latency"=30102456566 I1111 02:53:31.297901 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="f4a131ee-616b-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=30102565567 I1111 02:53:31.957538 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 02:53:31.957590 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:53:31.957619 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount with mount options([nouuid]) I1111 02:53:31.957631 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 135 lines ... I1111 02:53:55.166591 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:53:55.166604 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6jt2w-cloned","csi.storage.k8s.io/pvc/namespace":"multivolume-8159","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4"} I1111 02:53:55.166785 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:53:59.197353 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:53:59.197377 1 utils.go:79] GRPC request: {} I1111 02:53:59.197424 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:54:11.826122 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:54:11.826194 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="14cc9563-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4" "latency"=16659333518 I1111 02:54:11.826226 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="14cc9563-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16659452619 I1111 02:54:12.438084 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 02:54:12.438134 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:54:12.438163 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3ba666f2-fc79-47b2-8e01-9da74ac598e4/globalmount with mount options([nouuid]) I1111 02:54:12.438181 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 36 lines ... I1111 02:54:12.894071 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:54:12.894084 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-740cfa70-55fa-4647-975f-052c29c541b6/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-740cfa70-55fa-4647-975f-052c29c541b6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comnnhwr","csi.storage.k8s.io/pvc/namespace":"multivolume-7367","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-740cfa70-55fa-4647-975f-052c29c541b6"} I1111 02:54:12.894248 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-740cfa70-55fa-4647-975f-052c29c541b6-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:54:17.657075 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:54:17.657161 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:54:17.657182 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:54:17.657240 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:54:17.676414 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:54:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="223746f5-616c-11ed-ae2b-6045bd88ebdb" I1111 02:54:17.681436 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 4 milliseconds I1111 02:54:17.681637 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="223746f5-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5254842 I1111 02:54:17.730633 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:54:17.848142 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:54:18.425976 1 utils.go:78] GRPC call: /csi.v1.Node/NodeUnpublishVolume ... skipping 31 lines ... I1111 02:54:21.757252 1 nodeserver_v2.go:257] NodeUnstageVolume: unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount I1111 02:54:21.757282 1 mount_helper_common.go:99] "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount" is a mountpoint, unmounting I1111 02:54:21.757299 1 mount_linux.go:294] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount W1111 02:54:21.767838 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount" is not a mountpoint, deleting I1111 02:54:21.767914 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b78f5c34-fd0f-4e65-be44-ef642570638f/globalmount successfully I1111 02:54:21.767929 1 utils.go:85] GRPC response: {} I1111 02:54:24.525426 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:54:24.525502 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="1f5d9482-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-740cfa70-55fa-4647-975f-052c29c541b6" "latency"=11631171048 I1111 02:54:24.525537 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1f5d9482-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=11631301350 I1111 02:54:25.138312 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I1111 02:54:25.138360 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:54:25.138389 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-740cfa70-55fa-4647-975f-052c29c541b6/globalmount with mount options([nouuid]) I1111 02:54:25.138430 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 140 lines ... I1111 02:55:59.198339 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:55:59.198367 1 utils.go:79] GRPC request: {} I1111 02:55:59.198414 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:56:17.658617 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:56:17.658655 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:56:17.658711 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:56:17.658730 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:56:17.658763 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:56:17.676166 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:56:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="69bdc888-616c-11ed-ae2b-6045bd88ebdb" I1111 02:56:17.682222 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000001/status 200 OK in 5 milliseconds I1111 02:56:17.683690 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="69bdc888-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7531659 I1111 02:56:17.733067 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:56:17.851459 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:56:20.941438 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:56:20.941514 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5dacd066-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-52bced0b-008e-49ff-af52-1a26d04fa440" "latency"=23509166838 I1111 02:56:20.941554 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5dacd066-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=23509311040 I1111 02:56:20.944922 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:56:20.944992 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="5dacd30f-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1c0f985f-487a-4c3d-b317-db9166d07e68" "latency"=23512603665 I1111 02:56:20.945041 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="5dacd30f-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=23512727865 I1111 02:56:21.527299 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I1111 02:56:21.527350 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:56:21.527305 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sdd under /dev/disk/azure/scsi1/ I1111 02:56:21.527378 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1c0f985f-487a-4c3d-b317-db9166d07e68/globalmount with mount options([]) ... skipping 148 lines ... I1111 02:57:49.742174 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:57:49.742193 1 utils.go:79] GRPC request: {} I1111 02:57:49.742223 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:57:49.742875 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:57:49.742892 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268","csi.storage.k8s.io/pvc/name":"test.csi.azure.comd9fxt","csi.storage.k8s.io/pvc/namespace":"multivolume-1811","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268"} I1111 02:57:49.743064 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:57:54.114569 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:57:54.114642 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="a09e1567-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-db5d0648-7c4f-424b-8318-0c2c1ef85268" "latency"=4371485793 I1111 02:57:54.114675 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a09e1567-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4371618795 I1111 02:57:55.834047 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 02:57:55.834094 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:57:55.834112 1 utils.go:85] GRPC response: {} I1111 02:57:55.845058 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 83 lines ... I1111 02:59:11.238739 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:59:11.238755 1 utils.go:79] GRPC request: {} I1111 02:59:11.238791 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:59:11.239545 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:59:11.239561 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1e3dc67b-5da5-448f-a098-30547a547891/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1e3dc67b-5da5-448f-a098-30547a547891","csi.storage.k8s.io/pvc/name":"test.csi.azure.combnndp","csi.storage.k8s.io/pvc/namespace":"fsgroupchangepolicy-2932","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-1e3dc67b-5da5-448f-a098-30547a547891"} I1111 02:59:11.239727 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-1e3dc67b-5da5-448f-a098-30547a547891-k8s-agentpool1-30725374-vmss000001-attachment) I1111 02:59:15.221824 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:59:15.221900 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="d1317cd7-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1e3dc67b-5da5-448f-a098-30547a547891" "latency"=3982114445 I1111 02:59:15.221928 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d1317cd7-616c-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3982211547 I1111 02:59:16.909709 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 02:59:16.909759 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:59:16.909788 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1e3dc67b-5da5-448f-a098-30547a547891/globalmount with mount options([]) I1111 02:59:16.909797 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 169 lines ... I1111 03:00:34.834385 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 03:00:34.834403 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-645c9610-0f87-46c0-afce-8091ae8a2a72/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-645c9610-0f87-46c0-afce-8091ae8a2a72","csi.storage.k8s.io/pvc/name":"test.csi.azure.comv7sqt","csi.storage.k8s.io/pvc/namespace":"multivolume-1004","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-645c9610-0f87-46c0-afce-8091ae8a2a72"} I1111 03:00:34.834561 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 03:00:34.834605 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-645c9610-0f87-46c0-afce-8091ae8a2a72-k8s-agentpool1-30725374-vmss000001-attachment) I1111 03:00:34.834571 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77f3f92f-9208-4a3f-8daa-d489e3800757/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-77f3f92f-9208-4a3f-8daa-d489e3800757","csi.storage.k8s.io/pvc/name":"test.csi.azure.com5jgjc","csi.storage.k8s.io/pvc/namespace":"multivolume-1004","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-77f3f92f-9208-4a3f-8daa-d489e3800757"} I1111 03:00:34.834732 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-77f3f92f-9208-4a3f-8daa-d489e3800757-k8s-agentpool1-30725374-vmss000001-attachment) I1111 03:00:44.562890 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 03:00:44.562951 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="03050d50-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-645c9610-0f87-46c0-afce-8091ae8a2a72" "latency"=9728274011 I1111 03:00:44.562971 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="03050d50-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9728394611 I1111 03:00:44.568327 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 03:00:44.568399 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="030512a0-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-77f3f92f-9208-4a3f-8daa-d489e3800757" "latency"=9733603358 I1111 03:00:44.568438 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="030512a0-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9733716259 I1111 03:00:45.145814 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I1111 03:00:45.145869 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 03:00:45.145906 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77f3f92f-9208-4a3f-8daa-d489e3800757/globalmount with mount options([]) I1111 03:00:45.145917 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 163 lines ... I1111 03:02:56.526887 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 03:02:56.526904 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1beddae0-e39c-412c-b950-597e7ac0d449/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-1beddae0-e39c-412c-b950-597e7ac0d449","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-1beddae0-e39c-412c-b950-597e7ac0d449"} I1111 03:02:56.527101 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-1beddae0-e39c-412c-b950-597e7ac0d449-k8s-agentpool1-30725374-vmss000001-attachment) I1111 03:02:59.197838 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 03:02:59.197865 1 utils.go:79] GRPC request: {} I1111 03:02:59.197910 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 03:03:00.210517 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 03:03:00.210582 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="57799c70-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1beddae0-e39c-412c-b950-597e7ac0d449" "latency"=3683389763 I1111 03:03:00.210613 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="57799c70-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3683531864 I1111 03:03:01.895244 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 03:03:01.895308 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I1111 03:03:01.895337 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1beddae0-e39c-412c-b950-597e7ac0d449/globalmount with mount options([]) I1111 03:03:01.895355 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 62 lines ... I1111 03:03:57.738974 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 03:03:57.738989 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-29f57298-87bb-4966-8af3-4495005617ba/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-29f57298-87bb-4966-8af3-4495005617ba","csi.storage.k8s.io/pvc/name":"nginx-azuredisk-ephemeral-azuredisk01","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-29f57298-87bb-4966-8af3-4495005617ba"} I1111 03:03:57.739156 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-29f57298-87bb-4966-8af3-4495005617ba-k8s-agentpool1-30725374-vmss000001-attachment) I1111 03:03:59.197698 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 03:03:59.197722 1 utils.go:79] GRPC request: {} I1111 03:03:59.197766 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 03:04:07.793887 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 03:04:07.793958 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="7bf5d4bb-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-29f57298-87bb-4966-8af3-4495005617ba" "latency"=10054739161 I1111 03:04:07.793988 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="7bf5d4bb-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=10054843662 I1111 03:04:08.384698 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 03:04:08.384751 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I1111 03:04:08.384783 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-29f57298-87bb-4966-8af3-4495005617ba/globalmount with mount options([]) I1111 03:04:08.384802 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 42 lines ... I1111 03:04:17.782401 1 utils.go:79] GRPC request: {} I1111 03:04:17.782431 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 03:04:17.783263 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 03:04:17.783280 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cdaad550-6d6a-492a-a447-eaf824cd5686/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-cdaad550-6d6a-492a-a447-eaf824cd5686","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-hh9gg-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-cdaad550-6d6a-492a-a447-eaf824cd5686"} I1111 03:04:17.783444 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-cdaad550-6d6a-492a-a447-eaf824cd5686-k8s-agentpool1-30725374-vmss000001-attachment) I1111 03:04:17.861572 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 03:04:27.692091 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 03:04:27.692152 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000001" "disk.csi.azure.com/request-id"="87e8589b-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cdaad550-6d6a-492a-a447-eaf824cd5686" "latency"=9908619871 I1111 03:04:27.692184 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="87e8589b-616d-11ed-ae2b-6045bd88ebdb" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9908750872 I1111 03:04:28.291768 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I1111 03:04:28.291818 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType StandardSSD_LRS I1111 03:04:28.291850 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cdaad550-6d6a-492a-a447-eaf824cd5686/globalmount with mount options([]) I1111 03:04:28.291860 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 143 lines ... I1111 02:06:43.300895 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=2523&timeout=9m44s&timeoutSeconds=584&watch=true 200 OK in 1 milliseconds I1111 02:06:43.301758 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2332&timeout=6m22s&timeoutSeconds=382&watch=true 200 OK in 0 milliseconds I1111 02:06:43.391085 1 shared_informer.go:285] caches populated I1111 02:06:43.391111 1 azuredisk_v2.go:192] driver userAgent: test.csi.azure.com/latest-v2-9ac0207e3b71fdd2858caea5dad8b15c528821d0 e2e-test I1111 02:06:43.391135 1 azure_disk_utils.go:545] reading cloud config from secret kube-system/azure-cloud-provider I1111 02:06:43.394192 1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds I1111 02:06:43.394460 1 azure_disk_utils.go:552] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found I1111 02:06:43.394479 1 azure_disk_utils.go:557] could not read cloud config from secret kube-system/azure-cloud-provider I1111 02:06:43.394488 1 azure_disk_utils.go:567] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json I1111 02:06:43.394509 1 azure_disk_utils.go:575] read cloud config from file: /etc/kubernetes/azure.json successfully I1111 02:06:43.395201 1 azure_auth.go:245] Using AzurePublicCloud environment I1111 02:06:43.395247 1 azure_auth.go:96] azure: using managed identity extension to retrieve access token I1111 02:06:43.395270 1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token ... skipping 103 lines ... I1111 02:07:36.571411 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ea425b32-c65a-4964-9a98-97164117dec3-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:07:36.571516 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:07:40.322007 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:07:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9edcdd77-6165-11ed-9710-6045bd88e3ef" I1111 02:07:40.330148 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds I1111 02:07:40.330368 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9edcdd77-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8417184 I1111 02:07:43.085071 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:43.085171 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:07:43.085213 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:07:43.158495 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:43.239442 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:43.301981 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:43.418162 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:07:59.222991 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:07:59.223010 1 utils.go:79] GRPC request: {} I1111 02:07:59.223050 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:08:10.329736 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:08:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="b0bfaf02-6165-11ed-9710-6045bd88e3ef" I1111 02:08:10.340166 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 10 milliseconds I1111 02:08:10.340365 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="b0bfaf02-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10661832 I1111 02:08:13.087043 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:13.087208 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:08:13.087266 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:08:13.159295 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:13.240571 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:13.302096 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:13.418747 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:29.224994 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:08:29.225020 1 utils.go:79] GRPC request: {} I1111 02:08:29.225071 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:08:40.326722 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:08:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="c2a0dc87-6165-11ed-9710-6045bd88e3ef" I1111 02:08:40.335404 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds I1111 02:08:40.335574 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="c2a0dc87-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8873293 I1111 02:08:43.087472 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:43.087800 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:08:43.087924 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:08:43.159973 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:43.241352 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:43.302430 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:43.418906 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:08:59.223353 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:08:59.223373 1 utils.go:79] GRPC request: {} I1111 02:08:59.223447 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:09:10.319122 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:09:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="d48154bb-6165-11ed-9710-6045bd88e3ef" I1111 02:09:10.337125 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 17 milliseconds I1111 02:09:10.337549 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="d48154bb-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=18506304 I1111 02:09:13.087903 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:13.088129 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:09:13.088160 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:09:13.160529 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:13.241825 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:13.303254 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:13.419667 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:29.222979 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:09:29.223002 1 utils.go:79] GRPC request: {} I1111 02:09:29.223049 1 utils.go:85] GRPC response: {"ready":{"value":true}} E1111 02:09:36.565388 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9ca0936c-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea425b32-c65a-4964-9a98-97164117dec3" "latency"=119993476327 E1111 02:09:36.565565 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9ca0936c-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994114040 E1111 02:09:36.565389 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9ca0973d-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d" "latency"=119993532828 E1111 02:09:36.565587 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-ea425b32-c65a-4964-9a98-97164117dec3) to be attached to node (k8s-agentpool1-30725374-vmss000002): context canceled E1111 02:09:36.565636 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9ca0973d-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994124340 E1111 02:09:36.565669 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d) to be attached to node (k8s-agentpool1-30725374-vmss000002): context deadline exceeded I1111 02:09:37.162067 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:09:37.162093 1 utils.go:79] GRPC request: {} I1111 02:09:37.162136 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:09:37.162146 1 utils.go:79] GRPC request: {} I1111 02:09:37.162138 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:09:37.162190 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 16 lines ... I1111 02:09:37.167009 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d","csi.storage.k8s.io/pvc/name":"test.csi.azure.coms6mwg","csi.storage.k8s.io/pvc/namespace":"multivolume-5754","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d"} I1111 02:09:37.167162 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:09:40.321861 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:09:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="e66363b7-6165-11ed-9710-6045bd88e3ef" I1111 02:09:40.329717 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds I1111 02:09:40.329944 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="e66363b7-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8138271 I1111 02:09:43.090328 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:43.090452 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:09:43.090475 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:09:43.160759 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:43.242796 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:43.304091 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:43.420446 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:09:59.223193 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:09:59.223232 1 utils.go:79] GRPC request: {} I1111 02:09:59.223281 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:10:10.326088 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:10:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="f845ac8d-6165-11ed-9710-6045bd88e3ef" I1111 02:10:10.333498 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds I1111 02:10:10.333794 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="f845ac8d-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7732768 I1111 02:10:13.092505 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:13.092627 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:10:13.092683 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:10:13.160941 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:13.243294 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:13.304627 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:13.421033 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:29.222916 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:10:29.222941 1 utils.go:79] GRPC request: {} I1111 02:10:29.222988 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:10:40.323396 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:10:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="0a26e677-6166-11ed-9710-6045bd88e3ef" I1111 02:10:40.331732 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds I1111 02:10:40.331919 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="0a26e677-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8553185 I1111 02:10:43.095310 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:43.095611 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:10:43.095686 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:10:43.161573 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:43.243834 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:43.305262 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:43.421829 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:10:59.223591 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:10:59.223612 1 utils.go:79] GRPC request: {} I1111 02:10:59.223662 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:11:10.326446 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:11:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1c0900e3-6166-11ed-9710-6045bd88e3ef" I1111 02:11:10.334295 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds I1111 02:11:10.334448 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1c0900e3-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8031575 I1111 02:11:13.098204 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:13.098327 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:11:13.098341 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:11:13.162751 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:13.243945 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:13.306243 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:13.422721 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:11:29.223196 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:11:29.223233 1 utils.go:79] GRPC request: {} I1111 02:11:29.223283 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:11:32.719657 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:11:32.719723 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="e4820202-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ea425b32-c65a-4964-9a98-97164117dec3" "latency"=115552610284 I1111 02:11:32.719759 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e4820202-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=115552739286 I1111 02:11:32.719771 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:11:32.719846 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="e4820740-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-dff40eec-4c58-44e4-aee2-c1538eb6ec4d" "latency"=115552621585 I1111 02:11:32.719877 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="e4820740-6165-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=115552723887 I1111 02:11:33.311953 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 02:11:33.312014 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:11:33.312045 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ea425b32-c65a-4964-9a98-97164117dec3/globalmount with mount options([nouuid]) I1111 02:11:33.312067 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 266 lines ... I1111 02:12:42.398657 1 utils.go:79] GRPC request: {} I1111 02:12:42.398689 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:12:42.399414 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:12:42.399429 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f","csi.storage.k8s.io/pvc/name":"test.csi.azure.com6k9wb","csi.storage.k8s.io/pvc/namespace":"multivolume-7685","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f"} I1111 02:12:42.399599 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:12:43.102293 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:43.102417 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:12:43.102484 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:12:43.164662 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:43.245941 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:43.307381 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:43.423793 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:12:59.223293 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:12:59.223311 1 utils.go:79] GRPC request: {} I1111 02:12:59.223349 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:13:05.302389 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: Watch close - *v1.CustomResourceDefinition total 0 items received I1111 02:13:05.304753 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=2332&timeout=6m20s&timeoutSeconds=380&watch=true 200 OK in 2 milliseconds I1111 02:13:07.254435 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:13:07.254501 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="52bc2568-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-3337fe34-dd9f-4493-8a58-4f4fa00d239b" "latency"=25157018542 I1111 02:13:07.254524 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="52bc2568-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=25157119844 I1111 02:13:07.262631 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:13:07.262680 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="52ea40d4-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f" "latency"=24863031150 I1111 02:13:07.262704 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="52ea40d4-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24863129652 I1111 02:13:07.853579 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 02:13:07.853626 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:13:07.853653 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e8718e1c-528b-4fd6-a400-43aa6a13394f/globalmount with mount options([nouuid]) I1111 02:13:07.853666 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 119 lines ... I1111 02:13:33.380220 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext3"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40","csi.storage.k8s.io/pvc/name":"test.csi.azure.comn4n42","csi.storage.k8s.io/pvc/namespace":"volume-6500","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40"} I1111 02:13:33.380379 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:13:40.320209 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:13:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7570393f-6166-11ed-9710-6045bd88e3ef" I1111 02:13:40.328033 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds I1111 02:13:40.328336 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7570393f-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8234285 I1111 02:13:43.105229 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:43.105488 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:13:43.165581 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:43.246890 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:43.308316 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:43.424691 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:13:46.426301 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:13:46.426376 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="714d4df2-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40" "latency"=13045933037 I1111 02:13:46.426406 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="714d4df2-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13046038440 I1111 02:13:47.024198 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I1111 02:13:47.024247 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:13:47.024275 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e7215e08-5df4-4a62-9969-7a3c177b9e40/globalmount with mount options([]) I1111 02:13:47.024283 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 62 lines ... I1111 02:14:10.324282 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:14:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="87527c9e-6166-11ed-9710-6045bd88e3ef" I1111 02:14:10.333738 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 9 milliseconds I1111 02:14:10.333912 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="87527c9e-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9699413 I1111 02:14:12.419349 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 26 items received I1111 02:14:12.425273 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000002&resourceVersion=5743&timeout=6m23s&timeoutSeconds=383&watch=true 200 OK in 5 milliseconds I1111 02:14:13.106090 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:13.106248 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:14:13.159336 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 75 items received I1111 02:14:13.162081 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=5743&timeout=7m23s&timeoutSeconds=443&watch=true 200 OK in 2 milliseconds I1111 02:14:13.166427 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:13.247764 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:13.309026 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:13.425287 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:29.222810 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:14:29.222832 1 utils.go:79] GRPC request: {} I1111 02:14:29.222880 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:14:40.321004 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:14:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9933a079-6166-11ed-9710-6045bd88e3ef" I1111 02:14:40.327232 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 5 milliseconds I1111 02:14:40.327435 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9933a079-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6479541 I1111 02:14:43.106365 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:43.106462 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:14:43.167033 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:43.248349 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:43.309701 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:43.426244 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:14:57.386712 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:14:57.386772 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="83b5cd8d-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-98672851-fd48-4b89-9310-e3e3f222e658" "latency"=53122505119 I1111 02:14:57.386793 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="83b5cd8d-6166-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=53122602220 I1111 02:14:57.994043 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdc under /dev/disk/azure/scsi1/ I1111 02:14:57.994090 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I1111 02:14:57.994117 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-98672851-fd48-4b89-9310-e3e3f222e658/globalmount with mount options([]) I1111 02:14:57.994126 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3]) ... skipping 128 lines ... I1111 02:18:00.598857 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936","csi.storage.k8s.io/pvc/name":"test.csi.azure.commhdpp","csi.storage.k8s.io/pvc/namespace":"provisioning-3283","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936"} I1111 02:18:00.599036 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:18:10.319836 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:18:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="165ee779-6167-11ed-9710-6045bd88e3ef" I1111 02:18:10.328306 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds I1111 02:18:10.328941 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="165ee779-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9100299 I1111 02:18:13.109870 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:13.110180 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:18:13.172173 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:13.254533 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:13.316088 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:13.432535 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:18:24.818396 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:18:24.818468 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1093a2a4-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936" "latency"=24219373128 I1111 02:18:24.818497 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1093a2a4-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=24219478830 I1111 02:18:26.539430 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 02:18:26.539483 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:18:26.539513 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a3d9ddeb-2aba-4efa-a438-b77ee53d5936/globalmount with mount options([]) I1111 02:18:26.539523 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 125 lines ... I1111 02:21:14.709929 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:21:14.710635 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:21:14.710649 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a42f45dd-17b3-4305-8325-781135ac7133/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a42f45dd-17b3-4305-8325-781135ac7133","csi.storage.k8s.io/pvc/name":"test.csi.azure.comxjsrj","csi.storage.k8s.io/pvc/namespace":"multivolume-6620","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-a42f45dd-17b3-4305-8325-781135ac7133"} I1111 02:21:14.710828 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a42f45dd-17b3-4305-8325-781135ac7133-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:21:16.085586 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 311 items received I1111 02:21:16.093447 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=9697&timeout=5m31s&timeoutSeconds=331&watch=true 200 OK in 7 milliseconds I1111 02:21:28.634056 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:21:28.634124 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="8446be2f-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a42f45dd-17b3-4305-8325-781135ac7133" "latency"=13923233041 I1111 02:21:28.634161 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8446be2f-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13923346444 I1111 02:21:29.222822 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:21:29.222846 1 utils.go:79] GRPC request: {} I1111 02:21:29.222893 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:21:30.362473 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ ... skipping 88 lines ... I1111 02:22:16.031785 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-aaa672b4-088a-40de-bf83-32888502d696-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:22:29.224687 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:22:29.224712 1 utils.go:79] GRPC request: {} I1111 02:22:29.224760 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:22:31.243708 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 207 items received I1111 02:22:31.246889 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=10423&timeout=9m53s&timeoutSeconds=593&watch=true 200 OK in 2 milliseconds I1111 02:22:32.349762 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:22:32.349837 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="a8d3948d-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-aaa672b4-088a-40de-bf83-32888502d696" "latency"=16317989526 I1111 02:22:32.349867 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="a8d3948d-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=16318091929 I1111 02:22:34.068122 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I1111 02:22:34.068166 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:22:34.068197 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-aaa672b4-088a-40de-bf83-32888502d696/globalmount with mount options([]) I1111 02:22:34.068207 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 48 lines ... I1111 02:22:46.859071 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:22:46.859085 1 utils.go:79] GRPC request: {} I1111 02:22:46.859115 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:22:46.859888 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:22:46.859905 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82","csi.storage.k8s.io/pvc/name":"inline-volume-tester2-2wzwm-my-volume-0","csi.storage.k8s.io/pvc/namespace":"ephemeral-6566","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82"} I1111 02:22:46.860085 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:22:52.290793 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:22:52.290861 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="bb339a5d-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82" "latency"=5430710235 I1111 02:22:52.290886 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="bb339a5d-6167-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5430832638 I1111 02:22:52.900154 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ I1111 02:22:52.900202 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:22:52.900238 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cae9a0f6-2ca6-41b2-b348-493a8f4daa82/globalmount with mount options([]) I1111 02:22:52.900252 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 153 lines ... I1111 02:25:30.708508 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-65b99319-536b-4adb-acd1-8410dc77d4c5","csi.storage.k8s.io/pvc/name":"test.csi.azure.comhdrpr","csi.storage.k8s.io/pvc/namespace":"multivolume-8220","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5"} I1111 02:25:30.708729 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-65b99319-536b-4adb-acd1-8410dc77d4c5-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:25:40.321313 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:25:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="2297aeb2-6168-11ed-9710-6045bd88e3ef" I1111 02:25:40.329460 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds I1111 02:25:40.329690 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="2297aeb2-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8420785 I1111 02:25:43.126045 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:43.126210 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:25:43.126273 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:25:43.181364 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:43.265576 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:43.326002 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:43.443407 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:25:59.223417 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:25:59.223439 1 utils.go:79] GRPC request: {} I1111 02:25:59.223481 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:26:10.326425 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:26:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="347a19f2-6168-11ed-9710-6045bd88e3ef" I1111 02:26:10.337808 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 11 milliseconds I1111 02:26:10.338018 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="347a19f2-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11621653 I1111 02:26:13.126740 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:13.126906 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:13.126938 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:13.182052 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:13.266348 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:13.326860 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:13.444430 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:29.222953 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:26:29.222977 1 utils.go:79] GRPC request: {} I1111 02:26:29.223027 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:26:40.321362 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:26:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="465af610-6168-11ed-9710-6045bd88e3ef" I1111 02:26:40.329830 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds I1111 02:26:40.330059 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="465af610-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8754891 I1111 02:26:43.126983 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:43.127169 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:43.127280 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:26:43.182176 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:43.267487 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:43.327792 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:43.445243 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:26:47.094388 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 200 items received I1111 02:26:47.103549 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=12534&timeout=6m42s&timeoutSeconds=402&watch=true 200 OK in 8 milliseconds I1111 02:26:59.223301 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:26:59.223326 1 utils.go:79] GRPC request: {} I1111 02:26:59.223363 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:27:10.329034 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:27:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="583dc5f2-6168-11ed-9710-6045bd88e3ef" I1111 02:27:10.336000 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 6 milliseconds I1111 02:27:10.336200 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="583dc5f2-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7193757 I1111 02:27:13.128016 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:13.128350 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:13.128395 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:13.182299 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:13.267705 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:13.328084 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:13.445380 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:24.430287 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 22 items received I1111 02:27:24.433866 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dk8s-agentpool1-30725374-vmss000002&resourceVersion=12978&timeout=7m19s&timeoutSeconds=439&watch=true 200 OK in 3 milliseconds I1111 02:27:29.223721 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:27:29.223746 1 utils.go:79] GRPC request: {} I1111 02:27:29.223782 1 utils.go:85] GRPC response: {"ready":{"value":true}} E1111 02:27:30.501404 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cbe22da-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-09de6e98-ef97-4118-9359-0dcf58025169" "latency"=119994337430 E1111 02:27:30.501465 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1cbe22da-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119994513434 E1111 02:27:30.501483 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-09de6e98-ef97-4118-9359-0dcf58025169) to be attached to node (k8s-agentpool1-30725374-vmss000002): context canceled E1111 02:27:30.706013 1 conditionwaiter.go:50] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cdceca8-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-65b99319-536b-4adb-acd1-8410dc77d4c5" "latency"=119997140092 E1111 02:27:30.706095 1 crdprovisioner.go:760] "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1cdceca8-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=119997378297 E1111 02:27:30.706113 1 utils.go:83] GRPC error: rpc error: code = Internal desc = failed to wait for volume (/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5) to be attached to node (k8s-agentpool1-30725374-vmss000002): context deadline exceeded I1111 02:27:31.091833 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:27:31.091855 1 utils.go:79] GRPC request: {} I1111 02:27:31.091902 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:27:31.094866 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:27:31.094890 1 utils.go:79] GRPC request: {} I1111 02:27:31.094936 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} ... skipping 13 lines ... I1111 02:27:31.296685 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-65b99319-536b-4adb-acd1-8410dc77d4c5","csi.storage.k8s.io/pvc/name":"test.csi.azure.comhdrpr","csi.storage.k8s.io/pvc/namespace":"multivolume-8220","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-65b99319-536b-4adb-acd1-8410dc77d4c5"} I1111 02:27:31.296851 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-65b99319-536b-4adb-acd1-8410dc77d4c5-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:27:40.320999 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:27:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="6a1e2e2a-6168-11ed-9710-6045bd88e3ef" I1111 02:27:40.328169 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 6 milliseconds I1111 02:27:40.328438 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="6a1e2e2a-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7487063 I1111 02:27:43.131071 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:43.131191 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:43.131205 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:27:43.182507 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:43.268855 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:43.329259 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:43.445672 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:27:59.223625 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:27:59.223653 1 utils.go:79] GRPC request: {} I1111 02:27:59.223702 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:28:10.326706 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:28:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7c00b0ec-6168-11ed-9710-6045bd88e3ef" I1111 02:28:10.333882 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 7 milliseconds I1111 02:28:10.334461 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7c00b0ec-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=7777569 I1111 02:28:13.131519 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:13.131630 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:28:13.131644 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:28:13.182798 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:13.269287 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:13.329712 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:13.446295 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:28:16.303744 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:28:16.303814 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="64bd35c6-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-65b99319-536b-4adb-acd1-8410dc77d4c5" "latency"=45006901273 I1111 02:28:16.303839 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="64bd35c6-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=45007013376 I1111 02:28:16.306273 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:28:16.306335 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="649eb14f-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-09de6e98-ef97-4118-9359-0dcf58025169" "latency"=45209430088 I1111 02:28:16.306367 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="649eb14f-6168-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=45209534291 I1111 02:28:16.908805 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I1111 02:28:16.908861 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:28:16.908899 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-09de6e98-ef97-4118-9359-0dcf58025169/globalmount with mount options([]) I1111 02:28:16.908919 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 189 lines ... I1111 02:32:40.068042 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-06727e11-d7e2-4848-b27e-049c575ae4b3/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-06727e11-d7e2-4848-b27e-049c575ae4b3","csi.storage.k8s.io/pvc/name":"pvc-lsnxb","csi.storage.k8s.io/pvc/namespace":"provisioning-3371","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-06727e11-d7e2-4848-b27e-049c575ae4b3"} I1111 02:32:40.068202 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-06727e11-d7e2-4848-b27e-049c575ae4b3-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:32:40.319581 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:32:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cee5550-6169-11ed-9710-6045bd88e3ef" I1111 02:32:40.325103 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 5 milliseconds I1111 02:32:40.325317 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cee5550-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=5769326 I1111 02:32:43.139292 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:32:43.139389 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:32:43.189747 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:32:43.276144 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:32:43.335766 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:32:43.453085 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:32:59.222577 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:32:59.222598 1 utils.go:79] GRPC request: {} I1111 02:32:59.222644 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:32:59.630858 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:32:59.630941 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="1cc7fb1f-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-06727e11-d7e2-4848-b27e-049c575ae4b3" "latency"=19562669815 I1111 02:32:59.630965 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="1cc7fb1f-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19562777918 I1111 02:33:01.329684 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 02:33:01.329731 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:33:01.329758 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-06727e11-d7e2-4848-b27e-049c575ae4b3/globalmount with mount options([]) I1111 02:33:01.329767 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 86 lines ... I1111 02:33:50.265685 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:33:50.265760 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:33:50.265701 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c","csi.storage.k8s.io/pvc/name":"test.csi.azure.com2w94g","csi.storage.k8s.io/pvc/namespace":"multivolume-5107","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c"} I1111 02:33:50.265771 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6","csi.storage.k8s.io/pvc/name":"test.csi.azure.comnnjhh","csi.storage.k8s.io/pvc/namespace":"multivolume-5107","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6"} I1111 02:33:50.265853 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:33:50.265897 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:33:58.446198 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:33:58.446261 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="469f4ca8-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6" "latency"=8180314923 I1111 02:33:58.446283 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="469f4ca8-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8180398424 I1111 02:33:58.446528 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:33:58.446609 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="469f4ad4-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c" "latency"=8180678930 I1111 02:33:58.446644 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="469f4ad4-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8180804133 I1111 02:33:58.832316 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:33:58.832337 1 utils.go:79] GRPC request: {} I1111 02:33:58.832373 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:33:58.835736 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 111 lines ... I1111 02:34:12.137565 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-c4cfb065-dbf9-4356-ac29-6e2c98a476c6/globalmount successfully I1111 02:34:12.137579 1 utils.go:85] GRPC response: {} W1111 02:34:12.138833 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c/globalmount" is not a mountpoint, deleting I1111 02:34:12.138881 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-7bc382cc-fd5c-4786-a2b9-17cfd2b0d29c/globalmount successfully I1111 02:34:12.138889 1 utils.go:85] GRPC response: {} I1111 02:34:13.141425 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:34:13.141553 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:34:13.191684 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:34:13.278074 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:34:13.337395 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:34:13.454834 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:34:13.862789 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:34:13.862909 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="4bbb3c27-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-eacb38a6-1de7-4c1b-b1ec-64badef707a4" "latency"=15025234167 I1111 02:34:13.862960 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="4bbb3c27-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15025369170 I1111 02:34:14.473964 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I1111 02:34:14.474012 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:34:14.474044 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun2 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-eacb38a6-1de7-4c1b-b1ec-64badef707a4/globalmount with mount options([]) I1111 02:34:14.474054 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun2" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun2]) ... skipping 57 lines ... W1111 02:34:27.357906 1 mount_helper_common.go:133] Warning: "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-eacb38a6-1de7-4c1b-b1ec-64badef707a4/globalmount" is not a mountpoint, deleting I1111 02:34:27.357982 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-eacb38a6-1de7-4c1b-b1ec-64badef707a4/globalmount successfully I1111 02:34:27.357998 1 utils.go:85] GRPC response: {} I1111 02:34:29.223792 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:34:29.223815 1 utils.go:79] GRPC request: {} I1111 02:34:29.223857 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:34:31.717899 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:34:31.717957 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="59986e7d-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a" "latency"=9620292013 I1111 02:34:31.717981 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="59986e7d-6169-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9620393615 I1111 02:34:32.315771 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun3 by sdf under /dev/disk/azure/scsi1/ I1111 02:34:32.315822 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun3. perfProfile none accountType I1111 02:34:32.315853 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun3 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-26be61f3-f3d3-4f69-bf48-97e4e2a5d30a/globalmount with mount options([]) I1111 02:34:32.315862 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun3" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun3]) ... skipping 173 lines ... I1111 02:39:29.223296 1 utils.go:79] GRPC request: {} I1111 02:39:29.223343 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:39:40.324869 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:39:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="17460aef-616a-11ed-9710-6045bd88e3ef" I1111 02:39:40.333132 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds I1111 02:39:40.333319 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="17460aef-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8550786 I1111 02:39:43.152863 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:43.153045 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:39:43.201123 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:43.286644 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:43.343953 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:43.462253 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:39:54.865588 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:39:54.865679 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="09924135-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1ec298d8-dc40-4bbc-b1a4-f6d8c48e69b6" "latency"=37529480391 I1111 02:39:54.865738 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="09924135-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=37529621294 I1111 02:39:56.523433 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 02:39:56.523487 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:39:56.523527 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1ec298d8-dc40-4bbc-b1a4-f6d8c48e69b6/globalmount with mount options([]) I1111 02:39:56.523551 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 149 lines ... I1111 02:43:29.224024 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:43:40.319926 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:43:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="a6526299-616a-11ed-9710-6045bd88e3ef" I1111 02:43:40.328632 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 8 milliseconds I1111 02:43:40.328784 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="a6526299-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8955396 I1111 02:43:41.189429 1 reflector.go:559] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 83 items received I1111 02:43:41.191964 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=21753&timeout=6m37s&timeoutSeconds=397&watch=true 200 OK in 2 milliseconds I1111 02:43:42.830727 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:43:42.830801 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="9b931bdd-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-71243770-1990-468c-869a-16af87cb006e" "latency"=20541648497 I1111 02:43:42.830845 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="9b931bdd-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=20541778100 I1111 02:43:43.163926 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:43.206499 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:43.293831 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:43:43.349194 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync ... skipping 80 lines ... I1111 02:44:44.195841 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:44:44.195855 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f8425869-8e2d-42e7-855e-5535677289bc/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f8425869-8e2d-42e7-855e-5535677289bc","csi.storage.k8s.io/pvc/name":"pvc-snapshottable-tester-ghnh8-my-volume","csi.storage.k8s.io/pvc/namespace":"snapshotting-4372","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-f8425869-8e2d-42e7-855e-5535677289bc"} I1111 02:44:44.196001 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-f8425869-8e2d-42e7-855e-5535677289bc-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:44:59.223049 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:44:59.223076 1 utils.go:79] GRPC request: {} I1111 02:44:59.223120 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:45:00.014601 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:45:00.014664 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="cc651d37-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f8425869-8e2d-42e7-855e-5535677289bc" "latency"=15818605571 I1111 02:45:00.014685 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="cc651d37-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=15818700473 I1111 02:45:00.929076 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 02:45:00.929096 1 utils.go:79] GRPC request: {} I1111 02:45:00.929128 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 02:45:00.929907 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 66 lines ... I1111 02:45:05.168675 1 nodeserver_v2.go:262] NodeUnstageVolume: unmount /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f8425869-8e2d-42e7-855e-5535677289bc/globalmount successfully I1111 02:45:05.168700 1 utils.go:85] GRPC response: {} I1111 02:45:10.319657 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:45:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="dbf740c3-616a-11ed-9710-6045bd88e3ef" I1111 02:45:10.326206 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 6 milliseconds I1111 02:45:10.326408 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="dbf740c3-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=6853449 I1111 02:45:13.167042 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:13.167311 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:45:13.167398 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:45:13.209321 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:13.296696 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:13.352137 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:13.470582 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:45:20.363515 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:45:20.363588 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="d65eaad6-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-177f539f-785c-4890-b732-6a8a4198f060" "latency"=19432548778 I1111 02:45:20.363612 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d65eaad6-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19432661080 I1111 02:45:20.369407 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:45:20.369464 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="d66d1850-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c34630bc-09fb-41e2-93ae-9313570c8666" "latency"=19343883242 I1111 02:45:20.369500 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="d66d1850-616a-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=19343987144 I1111 02:45:20.970647 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun2 by sde under /dev/disk/azure/scsi1/ I1111 02:45:20.970698 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun2. perfProfile none accountType I1111 02:45:20.970713 1 utils.go:85] GRPC response: {} I1111 02:45:20.970783 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdd under /dev/disk/azure/scsi1/ ... skipping 234 lines ... I1111 02:47:27.858275 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 02:47:27.858287 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b","volume_capability":{"AccessType":{"Block":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b","csi.storage.k8s.io/pvc/name":"pvc-g9tk8","csi.storage.k8s.io/pvc/namespace":"provisioning-4550","requestedsizegib":"5","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b"} I1111 02:47:27.858442 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b-k8s-agentpool1-30725374-vmss000002-attachment) I1111 02:47:29.223566 1 utils.go:78] GRPC call: /csi.v1.Identity/Probe I1111 02:47:29.223594 1 utils.go:79] GRPC request: {} I1111 02:47:29.223634 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:47:31.252842 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:47:31.252901 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="2df20518-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-c7fa64df-0ec6-434e-aa00-c57cbe9c3e1b" "latency"=3394396810 I1111 02:47:31.252934 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="2df20518-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3394502913 I1111 02:47:33.029255 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 02:47:33.029302 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType I1111 02:47:33.029313 1 utils.go:85] GRPC response: {} I1111 02:47:33.037272 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 57 lines ... I1111 02:47:59.223998 1 utils.go:79] GRPC request: {} I1111 02:47:59.224045 1 utils.go:85] GRPC response: {"ready":{"value":true}} I1111 02:48:10.319160 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T02:48:10Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="4740ffda-616b-11ed-9710-6045bd88e3ef" I1111 02:48:10.337695 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 18 milliseconds I1111 02:48:10.339188 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="4740ffda-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=20088738 I1111 02:48:13.174925 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:13.175167 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 02:48:13.214238 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:13.301622 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:13.356939 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:13.474466 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 02:48:25.035881 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 02:48:25.035954 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="40245ff0-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-99a107cb-e783-49cd-af6b-31646664aaec" "latency"=26648444098 I1111 02:48:25.035982 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="40245ff0-616b-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=26648566601 I1111 02:48:26.794915 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdc under /dev/disk/azure/scsi1/ I1111 02:48:26.794967 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType I1111 02:48:26.794980 1 utils.go:85] GRPC response: {} I1111 02:48:26.804257 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities ... skipping 404 lines ... I1111 03:03:40.320865 1 azuredisk_v2.go:562] "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-11-11T03:03:40Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7193ff42-616d-11ed-9710-6045bd88e3ef" I1111 03:03:40.341532 1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/k8s-agentpool1-30725374-vmss000002/status 200 OK in 20 milliseconds I1111 03:03:40.341732 1 azuredisk_v2.go:549] "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="7193ff42-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=20964657 I1111 03:03:41.331687 1 reflector.go:559] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: Watch close - *v1.CustomResourceDefinition total 0 items received I1111 03:03:41.338140 1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions?allowWatchBookmarks=true&resourceVersion=27092&timeout=8m17s&timeoutSeconds=497&watch=true 200 OK in 6 milliseconds I1111 03:03:43.212389 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 03:03:43.212649 1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil> I1111 03:03:43.231642 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 03:03:43.319931 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 03:03:43.377543 1 reflector.go:281] k8s.io/apiextensions-apiserver/pkg/client/informers/externalversions/factory.go:117: forcing resync I1111 03:03:43.493093 1 reflector.go:281] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync I1111 03:03:43.732714 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 03:03:43.732791 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="703f88fe-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-1f84464b-1ed5-465c-8635-350592edd63c" "latency"=5643183529 I1111 03:03:43.732814 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="703f88fe-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5643305831 I1111 03:03:45.393473 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/ I1111 03:03:45.393529 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType StandardSSD_LRS I1111 03:03:45.393576 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-1f84464b-1ed5-465c-8635-350592edd63c/globalmount with mount options([]) I1111 03:03:45.393595 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0]) ... skipping 45 lines ... I1111 03:04:18.577314 1 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities I1111 03:04:18.577334 1 utils.go:79] GRPC request: {} I1111 03:04:18.577366 1 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]} I1111 03:04:18.578253 1 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume I1111 03:04:18.578272 1 utils.go:79] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-49387867-afa8-4df9-b167-425fff340720/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-49387867-afa8-4df9-b167-425fff340720","csi.storage.k8s.io/pvc/name":"daemonset-azuredisk-ephemeral-t8cxp-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1668132413341-8081-test.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-9juvoppw/providers/Microsoft.Compute/disks/pvc-49387867-afa8-4df9-b167-425fff340720"} I1111 03:04:18.578509 1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-49387867-afa8-4df9-b167-425fff340720-k8s-agentpool1-30725374-vmss000002-attachment) I1111 03:04:26.933884 1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil> I1111 03:04:26.933946 1 conditionwaiter.go:50] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="k8s-agentpool1-30725374-vmss000002" "disk.csi.azure.com/request-id"="8861a925-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-49387867-afa8-4df9-b167-425fff340720" "latency"=8355361283 I1111 03:04:26.933969 1 crdprovisioner.go:760] "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).NodeStageVolume" "disk.csi.azure.com/request-id"="8861a925-616d-11ed-9710-6045bd88e3ef" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8355496586 I1111 03:04:27.525413 1 nodeprovisioner_unix.go:250] azureDisk - found /dev/disk/azure/scsi1/lun1 by sdd under /dev/disk/azure/scsi1/ I1111 03:04:27.525467 1 nodeserver_v2.go:170] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun1. perfProfile none accountType StandardSSD_LRS I1111 03:04:27.525502 1 nodeserver_v2.go:211] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun1 and mounting at /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-49387867-afa8-4df9-b167-425fff340720/globalmount with mount options([]) I1111 03:04:27.525521 1 mount_linux.go:487] Attempting to determine if disk "/dev/disk/azure/scsi1/lun1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun1]) ... skipping 565 lines ... # HELP process_virtual_memory_bytes Virtual memory size in bytes. # TYPE process_virtual_memory_bytes gauge process_virtual_memory_bytes 7.77207808e+08 # HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes. # TYPE process_virtual_memory_max_bytes gauge process_virtual_memory_max_bytes 1.8446744073709552e+19 make: *** [Makefile:341: e2e-test] Error 1 2022/11/11 03:05:59 process.go:155: Step 'make e2e-test' finished in 1h10m0.136321513s 2022/11/11 03:05:59 aksengine_helpers.go:425: downloading /root/tmp3853355860/log-dump.sh from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2022/11/11 03:05:59 util.go:70: curl https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump.sh 2022/11/11 03:05:59 process.go:153: Running: chmod +x /root/tmp3853355860/log-dump.sh 2022/11/11 03:05:59 process.go:155: Step 'chmod +x /root/tmp3853355860/log-dump.sh' finished in 2.156435ms 2022/11/11 03:05:59 aksengine_helpers.go:425: downloading /root/tmp3853355860/log-dump-daemonset.yaml from https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/hack/log-dump/log-dump-daemonset.yaml ... skipping 75 lines ... ssh key file /root/.ssh/id_rsa does not exist. Exiting. 2022/11/11 03:06:58 process.go:155: Step 'bash -c /root/tmp3853355860/win-ci-logs-collector.sh kubetest-9juvoppw.westeurope.cloudapp.azure.com /root/tmp3853355860 /root/.ssh/id_rsa' finished in 3.627918ms 2022/11/11 03:06:58 aksengine.go:1141: Deleting resource group: kubetest-9juvoppw. 2022/11/11 03:14:08 process.go:96: Saved XML output to /logs/artifacts/junit_runner.xml. 2022/11/11 03:14:08 process.go:153: Running: bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}" 2022/11/11 03:14:09 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 271.712235ms 2022/11/11 03:14:09 main.go:328: Something went wrong: encountered 1 errors: [error during make e2e-test: exit status 2] + EXIT_VALUE=1 + set +o xtrace Cleaning up after docker in docker. ================================================================================ Cleaning up after docker e03554d77005 ... skipping 4 lines ...