This job view page is being replaced by Spyglass soon. Check out the new job view.
PRedreed: [V2] feat: enable configuration of KubeClient burst setting
ResultABORTED
Tests 0 failed / 25 succeeded
Started2022-09-22 04:06
Elapsed1h37m
Revision13e6788f07136215d431f7b74a3f674e5bf81bce
Refs 1526

No Test Failures!


Show 25 Passed Tests

Show 82 Skipped Tests

Error lines from build-log.txt

... skipping 117 lines ...
61e1f0faa119: Pull complete
3ec160d0cdd2: Pull complete
Digest: sha256:9f0cd916bbd3939813cd800ff9ffb3c20aa2178c21d6cfaead81a6b57d591ff2
Status: Downloaded newer image for k8sprow.azurecr.io/azuredisk-csi:latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad
k8sprow.azurecr.io/azuredisk-csi:latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad
docker pull k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad || make azdiskschedulerextender-all push-manifest-azdiskschedulerextender
Error response from daemon: manifest for k8sprow.azurecr.io/azdiskschedulerextender-csi:latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad not found: manifest unknown: manifest tagged by "latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
docker buildx rm container-builder || true
error: no builder "container-builder" found
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
# https://github.com/docker/buildx/issues/464#issuecomment-741507760
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64
Unable to find image 'tonistiigi/binfmt:latest' locally
... skipping 1319 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 773 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 226 lines ...
Sep 22 04:21:11.853: INFO: Found ClusterRoles; assuming RBAC is enabled.
STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-5035
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
I0922 04:21:12.336332   15445 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 22 04:21:12.451: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ktwcf" in namespace "azuredisk-5035" to be "Succeeded or Failed"
Sep 22 04:21:12.505: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Pending", Reason="", readiness=false. Elapsed: 54.176216ms
Sep 22 04:21:14.563: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Pending", Reason="", readiness=false. Elapsed: 2.112014589s
Sep 22 04:21:16.619: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Pending", Reason="", readiness=false. Elapsed: 4.167969508s
Sep 22 04:21:18.675: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Pending", Reason="", readiness=false. Elapsed: 6.223931084s
Sep 22 04:21:20.731: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Pending", Reason="", readiness=false. Elapsed: 8.279762591s
Sep 22 04:21:22.787: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Pending", Reason="", readiness=false. Elapsed: 10.335581878s
... skipping 4 lines ...
Sep 22 04:21:33.061: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Running", Reason="", readiness=true. Elapsed: 20.610493309s
Sep 22 04:21:35.117: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Running", Reason="", readiness=true. Elapsed: 22.665816604s
Sep 22 04:21:37.173: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Running", Reason="", readiness=true. Elapsed: 24.721949225s
Sep 22 04:21:39.229: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Running", Reason="", readiness=false. Elapsed: 26.778160292s
Sep 22 04:21:41.284: INFO: Pod "azuredisk-volume-tester-ktwcf": Phase="Succeeded", Reason="", readiness=false. Elapsed: 28.832846808s
STEP: Saw pod success
Sep 22 04:21:41.284: INFO: Pod "azuredisk-volume-tester-ktwcf" satisfied condition "Succeeded or Failed"
Sep 22 04:21:41.284: INFO: deleting Pod "azuredisk-5035"/"azuredisk-volume-tester-ktwcf"
Sep 22 04:21:41.391: INFO: Pod azuredisk-volume-tester-ktwcf has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-ktwcf in namespace azuredisk-5035
Sep 22 04:21:41.456: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "azuredisk-5035" for this suite.
... skipping 18 lines ...
I0922 04:21:42.202615   15445 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 22 04:21:42.432: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-r7bj2" in namespace "azuredisk-824" to be "Succeeded or Failed"
Sep 22 04:21:42.516: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Pending", Reason="", readiness=false. Elapsed: 83.595917ms
Sep 22 04:21:44.571: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Pending", Reason="", readiness=false. Elapsed: 2.138853335s
Sep 22 04:21:46.629: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Pending", Reason="", readiness=false. Elapsed: 4.197264554s
Sep 22 04:21:48.688: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Pending", Reason="", readiness=false. Elapsed: 6.2556792s
Sep 22 04:21:50.743: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Pending", Reason="", readiness=false. Elapsed: 8.310717071s
Sep 22 04:21:52.798: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Pending", Reason="", readiness=false. Elapsed: 10.365611198s
... skipping 23 lines ...
Sep 22 04:22:42.211: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Running", Reason="", readiness=true. Elapsed: 59.779220319s
Sep 22 04:22:44.267: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Running", Reason="", readiness=true. Elapsed: 1m1.83482344s
Sep 22 04:22:46.321: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Running", Reason="", readiness=false. Elapsed: 1m3.889323388s
Sep 22 04:22:48.378: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Running", Reason="", readiness=false. Elapsed: 1m5.945619178s
Sep 22 04:22:50.433: INFO: Pod "azuredisk-volume-tester-r7bj2": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m8.001085812s
STEP: Saw pod success
Sep 22 04:22:50.433: INFO: Pod "azuredisk-volume-tester-r7bj2" satisfied condition "Succeeded or Failed"
Sep 22 04:22:50.433: INFO: deleting Pod "azuredisk-824"/"azuredisk-volume-tester-r7bj2"
Sep 22 04:22:50.526: INFO: Pod azuredisk-volume-tester-r7bj2 has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-r7bj2 in namespace azuredisk-824
STEP: validating provisioned PV
STEP: checking the PV
... skipping 118 lines ...
STEP: creating a PVC
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 22 04:29:50.812: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-rnzck" in namespace "azuredisk-3843" to be "Succeeded or Failed"
Sep 22 04:29:50.874: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Pending", Reason="", readiness=false. Elapsed: 61.917756ms
Sep 22 04:29:52.927: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Pending", Reason="", readiness=false. Elapsed: 2.114928639s
Sep 22 04:29:54.981: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Pending", Reason="", readiness=false. Elapsed: 4.169027183s
Sep 22 04:29:57.034: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Pending", Reason="", readiness=false. Elapsed: 6.221811826s
Sep 22 04:29:59.089: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Pending", Reason="", readiness=false. Elapsed: 8.276459147s
Sep 22 04:30:01.141: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Pending", Reason="", readiness=false. Elapsed: 10.329026011s
... skipping 19 lines ...
Sep 22 04:30:42.197: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Running", Reason="", readiness=true. Elapsed: 51.384782399s
Sep 22 04:30:44.249: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Running", Reason="", readiness=false. Elapsed: 53.436691019s
Sep 22 04:30:46.302: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Running", Reason="", readiness=false. Elapsed: 55.489778229s
Sep 22 04:30:48.355: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Running", Reason="", readiness=false. Elapsed: 57.543185799s
Sep 22 04:30:50.408: INFO: Pod "azuredisk-volume-tester-rnzck": Phase="Succeeded", Reason="", readiness=false. Elapsed: 59.595272116s
STEP: Saw pod success
Sep 22 04:30:50.408: INFO: Pod "azuredisk-volume-tester-rnzck" satisfied condition "Succeeded or Failed"
Sep 22 04:30:50.408: INFO: deleting Pod "azuredisk-3843"/"azuredisk-volume-tester-rnzck"
Sep 22 04:30:50.513: INFO: Pod azuredisk-volume-tester-rnzck has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-rnzck in namespace azuredisk-3843
STEP: validating provisioned PV
STEP: checking the PV
... skipping 453 lines ...
STEP: checking the PVC
STEP: validating provisioned PV
STEP: checking the PV
STEP: setting up the pod
I0922 04:44:59.074388   15445 resource_setup.go:63] adding PV (pvc-0bcd181b-5d02-4a22-9f84-f0209e876bae) to pod ()
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 22 04:44:59.128: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-ppbv6" in namespace "azuredisk-8469" to be "Succeeded or Failed"
Sep 22 04:44:59.179: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Pending", Reason="", readiness=false. Elapsed: 51.879046ms
Sep 22 04:45:01.233: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Pending", Reason="", readiness=false. Elapsed: 2.104889577s
Sep 22 04:45:03.286: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Pending", Reason="", readiness=false. Elapsed: 4.157955251s
Sep 22 04:45:05.338: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Pending", Reason="", readiness=false. Elapsed: 6.210434204s
Sep 22 04:45:07.391: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Pending", Reason="", readiness=false. Elapsed: 8.263500241s
Sep 22 04:45:09.444: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Pending", Reason="", readiness=false. Elapsed: 10.316536618s
... skipping 19 lines ...
Sep 22 04:45:50.520: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Running", Reason="", readiness=true. Elapsed: 51.392862493s
Sep 22 04:45:52.572: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Running", Reason="", readiness=true. Elapsed: 53.444428213s
Sep 22 04:45:54.625: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Running", Reason="", readiness=false. Elapsed: 55.496970009s
Sep 22 04:45:56.677: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Running", Reason="", readiness=false. Elapsed: 57.549095791s
Sep 22 04:45:58.730: INFO: Pod "azuredisk-volume-tester-ppbv6": Phase="Succeeded", Reason="", readiness=false. Elapsed: 59.602747159s
STEP: Saw pod success
Sep 22 04:45:58.731: INFO: Pod "azuredisk-volume-tester-ppbv6" satisfied condition "Succeeded or Failed"
Sep 22 04:45:58.731: INFO: deleting Pod "azuredisk-8469"/"azuredisk-volume-tester-ppbv6"
Sep 22 04:45:58.830: INFO: Pod azuredisk-volume-tester-ppbv6 has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-ppbv6 in namespace azuredisk-8469
Sep 22 04:45:58.889: INFO: deleting PVC "azuredisk-8469"/"pvc-6qkm8"
Sep 22 04:45:58.889: INFO: Deleting PersistentVolumeClaim "pvc-6qkm8"
... skipping 46 lines ...
Sep 22 04:46:42.971: INFO: PersistentVolumeClaim pvc-j5nsp found but phase is Pending instead of Bound.
Sep 22 04:46:45.023: INFO: PersistentVolumeClaim pvc-j5nsp found and phase=Bound (4.181495621s)
STEP: checking the PVC
STEP: validating provisioned PV
STEP: checking the PV
STEP: deploying the pod
STEP: checking that the pods command exits with no error
Sep 22 04:46:45.187: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-jppqh" in namespace "azuredisk-4340" to be "Succeeded or Failed"
Sep 22 04:46:45.239: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Pending", Reason="", readiness=false. Elapsed: 52.000757ms
Sep 22 04:46:47.292: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.105344008s
Sep 22 04:46:49.345: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.158206772s
Sep 22 04:46:51.398: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.211264147s
Sep 22 04:46:53.451: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.264083599s
Sep 22 04:46:55.505: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.317860243s
... skipping 14 lines ...
Sep 22 04:47:26.300: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Pending", Reason="", readiness=false. Elapsed: 41.113404567s
Sep 22 04:47:28.353: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Running", Reason="", readiness=true. Elapsed: 43.165758456s
Sep 22 04:47:30.405: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Running", Reason="", readiness=false. Elapsed: 45.218190281s
Sep 22 04:47:32.458: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Running", Reason="", readiness=false. Elapsed: 47.270662373s
Sep 22 04:47:34.510: INFO: Pod "azuredisk-volume-tester-jppqh": Phase="Succeeded", Reason="", readiness=false. Elapsed: 49.323050496s
STEP: Saw pod success
Sep 22 04:47:34.510: INFO: Pod "azuredisk-volume-tester-jppqh" satisfied condition "Succeeded or Failed"
Sep 22 04:47:34.510: INFO: deleting Pod "azuredisk-4340"/"azuredisk-volume-tester-jppqh"
Sep 22 04:47:34.605: INFO: Pod azuredisk-volume-tester-jppqh has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-jppqh in namespace azuredisk-4340
Sep 22 04:47:34.668: INFO: deleting PVC "azuredisk-4340"/"pvc-j5nsp"
Sep 22 04:47:34.668: INFO: Deleting PersistentVolumeClaim "pvc-j5nsp"
... skipping 39 lines ...
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: setting up the pod
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 22 04:48:16.666: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-hkcq9" in namespace "azuredisk-101" to be "Succeeded or Failed"
Sep 22 04:48:16.724: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Pending", Reason="", readiness=false. Elapsed: 58.364816ms
Sep 22 04:48:18.777: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Pending", Reason="", readiness=false. Elapsed: 2.110992121s
Sep 22 04:48:20.829: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Pending", Reason="", readiness=false. Elapsed: 4.163034714s
Sep 22 04:48:22.882: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Pending", Reason="", readiness=false. Elapsed: 6.216160059s
Sep 22 04:48:24.935: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Pending", Reason="", readiness=false. Elapsed: 8.269700661s
Sep 22 04:48:26.988: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Pending", Reason="", readiness=false. Elapsed: 10.322290632s
... skipping 19 lines ...
Sep 22 04:49:08.053: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Running", Reason="", readiness=true. Elapsed: 51.387446633s
Sep 22 04:49:10.107: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Running", Reason="", readiness=false. Elapsed: 53.441130347s
Sep 22 04:49:12.160: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Running", Reason="", readiness=false. Elapsed: 55.494109598s
Sep 22 04:49:14.214: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Running", Reason="", readiness=false. Elapsed: 57.547856929s
Sep 22 04:49:16.266: INFO: Pod "azuredisk-volume-tester-hkcq9": Phase="Succeeded", Reason="", readiness=false. Elapsed: 59.599779395s
STEP: Saw pod success
Sep 22 04:49:16.266: INFO: Pod "azuredisk-volume-tester-hkcq9" satisfied condition "Succeeded or Failed"
Sep 22 04:49:16.266: INFO: deleting Pod "azuredisk-101"/"azuredisk-volume-tester-hkcq9"
Sep 22 04:49:16.363: INFO: Pod azuredisk-volume-tester-hkcq9 has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-hkcq9 in namespace azuredisk-101
STEP: validating provisioned PV
STEP: checking the PV
... skipping 41 lines ...
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: setting up the pod
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 22 04:49:58.544: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-kz9n4" in namespace "azuredisk-3393" to be "Succeeded or Failed"
Sep 22 04:49:58.598: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Pending", Reason="", readiness=false. Elapsed: 53.479136ms
Sep 22 04:50:00.650: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Pending", Reason="", readiness=false. Elapsed: 2.105568606s
Sep 22 04:50:02.706: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Pending", Reason="", readiness=false. Elapsed: 4.161793571s
Sep 22 04:50:04.758: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Pending", Reason="", readiness=false. Elapsed: 6.214138505s
Sep 22 04:50:06.811: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Pending", Reason="", readiness=false. Elapsed: 8.267369875s
Sep 22 04:50:08.864: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Pending", Reason="", readiness=false. Elapsed: 10.319839677s
... skipping 15 lines ...
Sep 22 04:50:41.712: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Pending", Reason="", readiness=false. Elapsed: 43.167797929s
Sep 22 04:50:43.764: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Running", Reason="", readiness=true. Elapsed: 45.219924673s
Sep 22 04:50:45.816: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Running", Reason="", readiness=false. Elapsed: 47.272051584s
Sep 22 04:50:47.869: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Running", Reason="", readiness=false. Elapsed: 49.324491093s
Sep 22 04:50:49.921: INFO: Pod "azuredisk-volume-tester-kz9n4": Phase="Succeeded", Reason="", readiness=false. Elapsed: 51.377245555s
STEP: Saw pod success
Sep 22 04:50:49.921: INFO: Pod "azuredisk-volume-tester-kz9n4" satisfied condition "Succeeded or Failed"
Sep 22 04:50:49.921: INFO: deleting Pod "azuredisk-3393"/"azuredisk-volume-tester-kz9n4"
Sep 22 04:50:50.019: INFO: Pod azuredisk-volume-tester-kz9n4 has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-kz9n4 in namespace azuredisk-3393
STEP: validating provisioned PV
STEP: checking the PV
... skipping 116 lines ...
STEP: checking the PVC
STEP: validating provisioned PV
STEP: checking the PV
STEP: setting up the pod
I0922 04:52:19.227408   15445 resource_setup.go:63] adding PV (pvc-9c355c2c-8bb3-4463-9031-96824e50d877) to pod ()
STEP: deploying the pod
STEP: checking that the pod's command exits with an error
Sep 22 04:52:19.283: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-7fj2d" in namespace "azuredisk-3628" to be "Error status code"
Sep 22 04:52:19.334: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Pending", Reason="", readiness=false. Elapsed: 51.199157ms
Sep 22 04:52:21.386: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Pending", Reason="", readiness=false. Elapsed: 2.103727182s
Sep 22 04:52:23.439: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Pending", Reason="", readiness=false. Elapsed: 4.156709196s
Sep 22 04:52:25.493: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Pending", Reason="", readiness=false. Elapsed: 6.209896597s
Sep 22 04:52:27.545: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Pending", Reason="", readiness=false. Elapsed: 8.261947476s
Sep 22 04:52:29.598: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Pending", Reason="", readiness=false. Elapsed: 10.315280088s
... skipping 27 lines ...
Sep 22 04:53:27.075: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Pending", Reason="", readiness=false. Elapsed: 1m7.792374945s
Sep 22 04:53:29.127: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Running", Reason="", readiness=true. Elapsed: 1m9.844714103s
Sep 22 04:53:31.179: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Running", Reason="", readiness=true. Elapsed: 1m11.89658931s
Sep 22 04:53:33.232: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Running", Reason="", readiness=false. Elapsed: 1m13.9491864s
Sep 22 04:53:35.285: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.002430794s
Sep 22 04:53:37.340: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Running", Reason="", readiness=false. Elapsed: 1m18.057044875s
Sep 22 04:53:39.392: INFO: Pod "azuredisk-volume-tester-7fj2d": Phase="Failed", Reason="", readiness=false. Elapsed: 1m20.109137979s
STEP: Saw pod failure
Sep 22 04:53:39.392: INFO: Pod "azuredisk-volume-tester-7fj2d" satisfied condition "Error status code"
STEP: checking that pod logs contain expected message
Sep 22 04:53:39.498: INFO: deleting Pod "azuredisk-3628"/"azuredisk-volume-tester-7fj2d"
Sep 22 04:53:39.558: INFO: Pod azuredisk-volume-tester-7fj2d has the following logs: out-file : Access to the path 'C:\mnt\test-1\data' is denied.
At line:1 char:1
+ echo $null >> C:\mnt\test-1\data
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 458 lines ...
STEP: checking the PVC
STEP: validating provisioned PV
STEP: checking the PV
STEP: setting up the pod
I0922 05:01:18.399507   15445 resource_setup.go:63] adding PV (pvc-db6308c8-8c55-475a-af7d-6c1ef378fa00) to pod ()
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 22 05:01:18.454: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-dc85m" in namespace "azuredisk-4627" to be "Succeeded or Failed"
Sep 22 05:01:18.506: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Pending", Reason="", readiness=false. Elapsed: 51.866699ms
Sep 22 05:01:20.562: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Pending", Reason="", readiness=false. Elapsed: 2.107769931s
Sep 22 05:01:22.617: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Pending", Reason="", readiness=false. Elapsed: 4.163146999s
Sep 22 05:01:24.670: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Pending", Reason="", readiness=false. Elapsed: 6.215483708s
Sep 22 05:01:26.722: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Pending", Reason="", readiness=false. Elapsed: 8.267693862s
Sep 22 05:01:28.775: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Pending", Reason="", readiness=false. Elapsed: 10.320577208s
... skipping 29 lines ...
Sep 22 05:02:30.387: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Running", Reason="", readiness=true. Elapsed: 1m11.932882032s
Sep 22 05:02:32.442: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Running", Reason="", readiness=true. Elapsed: 1m13.988031755s
Sep 22 05:02:34.495: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Running", Reason="", readiness=false. Elapsed: 1m16.040652398s
Sep 22 05:02:36.553: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Running", Reason="", readiness=false. Elapsed: 1m18.098629029s
Sep 22 05:02:38.606: INFO: Pod "azuredisk-volume-tester-dc85m": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m20.151703687s
STEP: Saw pod success
Sep 22 05:02:38.606: INFO: Pod "azuredisk-volume-tester-dc85m" satisfied condition "Succeeded or Failed"
Sep 22 05:02:38.606: INFO: deleting Pod "azuredisk-4627"/"azuredisk-volume-tester-dc85m"
Sep 22 05:02:38.720: INFO: Pod azuredisk-volume-tester-dc85m has the following logs: hello world
hello world
hello world

STEP: Deleting pod azuredisk-volume-tester-dc85m in namespace azuredisk-4627
... skipping 155 lines ...
STEP: creating a PVC
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 22 05:03:45.464: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-dp46t" in namespace "azuredisk-7931" to be "Succeeded or Failed"
Sep 22 05:03:45.527: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Pending", Reason="", readiness=false. Elapsed: 63.384397ms
Sep 22 05:03:47.583: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Pending", Reason="", readiness=false. Elapsed: 2.119333447s
Sep 22 05:03:49.636: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Pending", Reason="", readiness=false. Elapsed: 4.17277458s
Sep 22 05:03:51.690: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Pending", Reason="", readiness=false. Elapsed: 6.226307603s
Sep 22 05:03:53.742: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Pending", Reason="", readiness=false. Elapsed: 8.278413676s
Sep 22 05:03:55.795: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Pending", Reason="", readiness=false. Elapsed: 10.33145553s
... skipping 20 lines ...
Sep 22 05:04:38.907: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Running", Reason="", readiness=true. Elapsed: 53.443812377s
Sep 22 05:04:40.960: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Running", Reason="", readiness=true. Elapsed: 55.496169885s
Sep 22 05:04:43.012: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Running", Reason="", readiness=false. Elapsed: 57.548660071s
Sep 22 05:04:45.065: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Running", Reason="", readiness=false. Elapsed: 59.601796906s
Sep 22 05:04:47.119: INFO: Pod "azuredisk-volume-tester-dp46t": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m1.655792777s
STEP: Saw pod success
Sep 22 05:04:47.119: INFO: Pod "azuredisk-volume-tester-dp46t" satisfied condition "Succeeded or Failed"
Sep 22 05:04:47.119: INFO: deleting Pod "azuredisk-7931"/"azuredisk-volume-tester-dp46t"
Sep 22 05:04:47.217: INFO: Pod azuredisk-volume-tester-dp46t has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-dp46t in namespace azuredisk-7931
STEP: validating provisioned PV
STEP: checking the PV
... skipping 367 lines ...
STEP: begin to delete the pod
Sep 22 05:16:14.768: INFO: deleting Pod "azuredisk-3863"/"azuredisk-volume-tester-7gh4j"
Sep 22 05:16:14.905: INFO: Pod azuredisk-volume-tester-7gh4j has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-7gh4j in namespace azuredisk-3863
Sep 22 05:16:47.078: INFO: deleting Pod "azuredisk-3863"/"azuredisk-volume-tester-7gh4j"
Sep 22 05:16:47.133: INFO: Error getting logs for pod azuredisk-volume-tester-7gh4j: the server could not find the requested resource (get pods azuredisk-volume-tester-7gh4j)
STEP: Deleting pod azuredisk-volume-tester-7gh4j in namespace azuredisk-3863
STEP: validating provisioned PV
STEP: checking the PV
Sep 22 05:16:47.299: INFO: deleting PVC "azuredisk-3863"/"pvc-tl6k9"
Sep 22 05:16:47.299: INFO: Deleting PersistentVolumeClaim "pvc-tl6k9"
STEP: waiting for claim's PV "pvc-5ca83ddd-99e4-4922-b4f6-def8bd2352c4" to be deleted
... skipping 43 lines ...
STEP: begin to delete the pod
Sep 22 05:17:38.456: INFO: deleting Pod "azuredisk-741"/"azuredisk-volume-tester-rxddk"
Sep 22 05:17:38.547: INFO: Pod azuredisk-volume-tester-rxddk has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-rxddk in namespace azuredisk-741
Sep 22 05:17:56.720: INFO: deleting Pod "azuredisk-741"/"azuredisk-volume-tester-rxddk"
Sep 22 05:17:56.774: INFO: Error getting logs for pod azuredisk-volume-tester-rxddk: the server could not find the requested resource (get pods azuredisk-volume-tester-rxddk)
STEP: Deleting pod azuredisk-volume-tester-rxddk in namespace azuredisk-741
STEP: validating provisioned PV
STEP: checking the PV
Sep 22 05:17:56.940: INFO: deleting PVC "azuredisk-741"/"pvc-7hsct"
Sep 22 05:17:56.940: INFO: Deleting PersistentVolumeClaim "pvc-7hsct"
STEP: waiting for claim's PV "pvc-fc274787-ab7a-4751-9d73-f934d576cb50" to be deleted
... skipping 524 lines ...

    test case not supported by Windows clusters

    /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26
------------------------------
Dynamic Provisioning [multi-az] 
  should check failed replica attachments are recreated after space is made from a volume detaching.
  /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1764
STEP: Creating a kubernetes client
Sep 22 05:25:19.118: INFO: >>> kubeConfig: /root/tmp1020537157/kubeconfig/kubeconfig.westus2.json
STEP: Building a namespace api object, basename azuredisk
STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-9454
STEP: Waiting for a default service account to be provisioned in namespace
... skipping 8 lines ...

S [SKIPPING] [1.493 seconds]
Dynamic Provisioning
/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40
  [multi-az]
  /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:48
    should check failed replica attachments are recreated after space is made from a volume detaching. [It]
    /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1764

    Skip flakey test until #1378 is fixed.

    /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1773
------------------------------
... skipping 103 lines ...
I0922 04:20:02.787031       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1512&timeout=7m27s&timeoutSeconds=447&watch=true 200 OK in 2 milliseconds
I0922 04:20:02.787859       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=1504&timeout=7m30s&timeoutSeconds=450&watch=true 200 OK in 1 milliseconds
I0922 04:20:02.881168       1 shared_informer.go:285] caches populated
I0922 04:20:02.881315       1 azuredisk_v2.go:193] driver userAgent: disk.csi.azure.com/latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad e2e-test
I0922 04:20:02.881419       1 azure_disk_utils.go:543] reading cloud config from secret kube-system/azure-cloud-provider
I0922 04:20:02.884790       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0922 04:20:02.885178       1 azure_disk_utils.go:550] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0922 04:20:02.885330       1 azure_disk_utils.go:555] could not read cloud config from secret kube-system/azure-cloud-provider
I0922 04:20:02.885354       1 azure_disk_utils.go:565] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0922 04:20:02.885389       1 azure_disk_utils.go:573] read cloud config from file: /etc/kubernetes/azure.json successfully
I0922 04:20:02.886260       1 azure_auth.go:245] Using AzurePublicCloud environment
I0922 04:20:02.886294       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I0922 04:20:02.886303       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
I0922 04:20:02.886463       1 azure_auth.go:113] azure: User Assigned MSI ID is client ID. Resource ID parsing error: %+vparsing failed for d55c9a9f-4cdb-4884-bfff-b50f401b2560. Invalid resource Id format
I0922 04:20:02.886618       1 azure.go:781] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000
I0922 04:20:02.886774       1 azure_interfaceclient.go:72] Azure InterfacesClient (read ops) using rate limit config: QPS=100, bucket=1000
I0922 04:20:02.886795       1 azure_interfaceclient.go:75] Azure InterfacesClient (write ops) using rate limit config: QPS=100, bucket=1000
I0922 04:20:02.886893       1 azure_vmsizeclient.go:68] Azure VirtualMachineSizesClient (read ops) using rate limit config: QPS=100, bucket=1000
I0922 04:20:02.886912       1 azure_vmsizeclient.go:71] Azure VirtualMachineSizesClient (write ops) using rate limit config: QPS=100, bucket=1000
I0922 04:20:02.886945       1 azure_storageaccountclient.go:70] Azure StorageAccountClient (read ops) using rate limit config: QPS=100, bucket=1000
... skipping 26 lines ...
I0922 04:20:02.888206       1 skus.go:123] NewNodeInfo: Starting to populate node and disk sku information.
I0922 04:20:02.988267       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.099932155 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:20:02.988306       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:20:02.988329       1 azure_vmss_cache.go:332] Node  has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:20:03.027187       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.038811369 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:20:03.027225       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
E0922 04:20:03.027252       1 azuredisk_v2.go:240] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance
I0922 04:20:03.027609       1 mount_linux.go:208] Detected OS without systemd
I0922 04:20:03.027743       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0922 04:20:03.027857       1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0922 04:20:03.027950       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0922 04:20:03.028046       1 driver.go:81] Enabling controller service capability: CLONE_VOLUME
I0922 04:20:03.028154       1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
... skipping 282 lines ...
I0922 04:20:03.568991       1 shared_state.go:483]  "msg"="Storing pod kube-scheduler-k8s-master-37910252-vmss000000 and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48"
I0922 04:20:03.569023       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/kube-scheduler-k8s-master-37910252-vmss000000." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48" "disk.csi/azure.com/pod-name"="kube-system/kube-scheduler-k8s-master-37910252-vmss000000"
I0922 04:20:03.569046       1 shared_state.go:307]  "msg"="Getting requested volumes for pod (kube-system/kube-scheduler-k8s-master-37910252-vmss000000)." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48" "disk.csi/azure.com/pod-name"="kube-system/kube-scheduler-k8s-master-37910252-vmss000000"
I0922 04:20:03.569106       1 pod.go:99]  "msg"="Pod kube-system/kube-scheduler-k8s-master-37910252-vmss000000 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48" "disk.csi/azure.com/pod-name"="kube-system/kube-scheduler-k8s-master-37910252-vmss000000"
I0922 04:20:03.569148       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/kube-scheduler-k8s-master-37910252-vmss000000" "latency"=117006
I0922 04:20:03.569186       1 shared_state.go:413]  "msg"="Adding pod csi-azuredisk-controller-6956ccfdfb-5jfsx to shared map with keyName kube-system/csi-azuredisk-controller-6956ccfdfb-5jfsx." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48"
I0922 04:20:03.571154       1 shared_state.go:419]  "msg"="Pod spec of pod csi-azuredisk-controller-6956ccfdfb-5jfsx is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-czmgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=15s --leader-election --leader-election-namespace=kube-system --worker-threads=40 --extra-create-metadata=true --strict-topology=true] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000a650c0 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-master-37910252-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000a650c8} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000a650d0}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000a650d8 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000a650dc PreemptionPolicy:0xc000798ea0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-czmgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48"
I0922 04:20:03.571237       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48"
I0922 04:20:03.571285       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48"
I0922 04:20:03.571389       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48"
I0922 04:20:03.571493       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {kube-api-access-czmgf {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48"
I0922 04:20:03.571564       1 shared_state.go:483]  "msg"="Storing pod csi-azuredisk-controller-6956ccfdfb-5jfsx and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48"
I0922 04:20:03.571602       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-azuredisk-controller-6956ccfdfb-5jfsx." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-controller-6956ccfdfb-5jfsx"
... skipping 101 lines ...
I0922 04:20:03.604971       1 shared_informer.go:285] caches populated
I0922 04:20:03.605003       1 controller.go:227] AzDiskControllerManager "msg"="Starting workers" "controller"="pv" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller" "worker count"=10
I0922 04:20:03.610085       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/azvolumes.disk.csi.azure.com/status 200 OK in 12 milliseconds
I0922 04:20:03.614562       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/azvolumeattachments.disk.csi.azure.com 200 OK in 3 milliseconds
I0922 04:20:03.624524       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/azvolumeattachments.disk.csi.azure.com/status 200 OK in 8 milliseconds
I0922 04:20:03.625692       1 azuredisk_v2.go:415]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d4b68d66-3a2d-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).StartControllersAndDieOnExit.func1" "latency"=123042082
I0922 04:20:03.640416       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="3791k8s00000001" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0922 04:20:03.640639       1 shared_state.go:413]  "msg"="Adding pod kube-apiserver-k8s-master-37910252-vmss000000 to shared map with keyName kube-system/kube-apiserver-k8s-master-37910252-vmss000000." 
I0922 04:20:03.640895       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="3791k8s00000002" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0922 04:20:03.641107       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d4cba77d-3a2d-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8201
I0922 04:20:03.641329       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="3791k8s00000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0922 04:20:03.641525       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d4cbb883-3a2d-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=10300
I0922 04:20:03.641740       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="3791k8s00000003" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0922 04:20:03.641827       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d4cbc887-3a2d-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=7500
I0922 04:20:03.642042       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-master-37910252-vmss000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0922 04:20:03.642234       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d4cbd46d-3a2d-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=7900
I0922 04:20:03.642474       1 shared_state.go:413]  "msg"="Adding pod csi-azuredisk-scheduler-extender-646b7857b-mhpvx to shared map with keyName kube-system/csi-azuredisk-scheduler-extender-646b7857b-mhpvx." 
I0922 04:20:03.642680       1 shared_state.go:419]  "msg"="Pod spec of pod kube-apiserver-k8s-master-37910252-vmss000000 is: {Volumes:[{Name:etc-kubernetes VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:etc-ssl-certs VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/ssl/certs,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-lib-kubelet VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/kubelet,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:msi VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/waagent/ManagedIdentity-Settings,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:sock VolumeSource:{HostPath:&HostPathVolumeSource{Path:/opt,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:auditlog VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/log/kubeaudit,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:kube-apiserver Image:mcr.microsoft.com/oss/kubernetes/kube-apiserver:v1.23.10 Command:[kube-apiserver] Args:[--advertise-address=10.240.0.4 --allow-privileged=true --anonymous-auth=false --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/log/kubeaudit/audit.log --audit-policy-file=/etc/kubernetes/addons/audit-policy.yaml --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --client-ca-file=/etc/kubernetes/certs/ca.crt --cloud-config=/etc/kubernetes/azure.json --cloud-provider=azure --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,ValidatingAdmissionWebhook,ResourceQuota,ExtendedResourceToleration,PodSecurityPolicy --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/certs/ca.crt --etcd-certfile=/etc/kubernetes/certs/etcdclient.crt --etcd-keyfile=/etc/kubernetes/certs/etcdclient.key --etcd-servers=https://127.0.0.1:2379 --feature-gates= --kubelet-client-certificate=/etc/kubernetes/certs/client.crt --kubelet-client-key=/etc/kubernetes/certs/client.key --profiling=false --proxy-client-cert-file=/etc/kubernetes/certs/proxy.crt --proxy-client-key-file=/etc/kubernetes/certs/proxy.key --requestheader-allowed-names= --requestheader-client-ca-file=/etc/kubernetes/certs/proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/certs/apiserver.key --service-account-lookup=true --service-account-signing-key-file=/etc/kubernetes/certs/apiserver.key --service-cluster-ip-range=10.0.0.0/16 --storage-backend=etcd3 --tls-cert-file=/etc/kubernetes/certs/apiserver.crt --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA --tls-private-key-file=/etc/kubernetes/certs/apiserver.key --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:etc-kubernetes ReadOnly:false MountPath:/etc/kubernetes SubPath: MountPropagation:<nil> SubPathExpr:} {Name:etc-ssl-certs ReadOnly:false MountPath:/etc/ssl/certs SubPath: MountPropagation:<nil> SubPathExpr:} {Name:var-lib-kubelet ReadOnly:false MountPath:/var/lib/kubelet SubPath: MountPropagation:<nil> SubPathExpr:} {Name:msi ReadOnly:true MountPath:/var/lib/waagent/ManagedIdentity-Settings SubPath: MountPropagation:<nil> SubPathExpr:} {Name:sock ReadOnly:false MountPath:/opt SubPath: MountPropagation:<nil> SubPathExpr:} {Name:auditlog ReadOnly:false MountPath:/var/log/kubeaudit SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000a8bce8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken:<nil> NodeName:k8s-master-37910252-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key: Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>}] HostAliases:[] PriorityClassName:system-node-critical Priority:0xc000a8bcf0 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000a8bcf4 PreemptionPolicy:0xc000ce8ab0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:etc-kubernetes VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:etc-ssl-certs VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/ssl/certs,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:var-lib-kubelet VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/kubelet,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:msi VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/lib/waagent/ManagedIdentity-Settings,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:sock VolumeSource:{HostPath:&HostPathVolumeSource{Path:/opt,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:auditlog VolumeSource:{HostPath:&HostPathVolumeSource{Path:/var/log/kubeaudit,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0922 04:20:03.642924       1 shared_state.go:457]  "msg"="Pod kube-apiserver-k8s-master-37910252-vmss000000: Skipping Volume {etc-kubernetes {&HostPathVolumeSource{Path:/etc/kubernetes,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.643341       1 shared_state.go:457]  "msg"="Pod kube-apiserver-k8s-master-37910252-vmss000000: Skipping Volume {etc-ssl-certs {&HostPathVolumeSource{Path:/etc/ssl/certs,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.643734       1 shared_state.go:457]  "msg"="Pod kube-apiserver-k8s-master-37910252-vmss000000: Skipping Volume {var-lib-kubelet {&HostPathVolumeSource{Path:/var/lib/kubelet,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
... skipping 117 lines ...
I0922 04:20:03.656980       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-node-win-hzhdw: Skipping Volume {registration-dir {&HostPathVolumeSource{Path:C:\\var\\lib\\kubelet\\plugins_registry\\,Type:*Directory,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.657041       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-node-win-hzhdw: Skipping Volume {kubelet-dir {&HostPathVolumeSource{Path:C:\\var\\lib\\kubelet\\,Type:*Directory,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.657092       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-node-win-hzhdw: Skipping Volume {plugin-dir {&HostPathVolumeSource{Path:C:\\var\\lib\\kubelet\\plugins\\disk.csi.azure.com\\,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.657140       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-node-win-hzhdw: Skipping Volume {azure-config {&HostPathVolumeSource{Path:C:\\k,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.657401       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-node-win-hzhdw: Skipping Volume {kube-api-access-wscl5 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.657494       1 shared_state.go:483]  "msg"="Storing pod csi-azuredisk-node-win-hzhdw and claim [] to podToClaimsMap map." 
I0922 04:20:03.647490       1 shared_state.go:419]  "msg"="Pod spec of pod csi-azuredisk-controller-6956ccfdfb-5jfsx is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-czmgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=15s --leader-election --leader-election-namespace=kube-system --worker-threads=40 --extra-create-metadata=true --strict-topology=true] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000d27ee0 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-master-37910252-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000d27ee8} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000d27ef0}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000d27ef8 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000d27efc PreemptionPolicy:0xc000d297f0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-czmgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0922 04:20:03.657753       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.657952       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.658216       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.658538       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {kube-api-access-czmgf {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.658674       1 shared_state.go:483]  "msg"="Storing pod csi-azuredisk-controller-6956ccfdfb-5jfsx and claim [] to podToClaimsMap map." 
I0922 04:20:03.645955       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/kube-apiserver-k8s-master-37910252-vmss000000." "disk.csi.azure.com/request-id"="d4cc6b17-3a2d-11ed-a99d-00224878ec48" "disk.csi/azure.com/pod-name"="kube-system/kube-apiserver-k8s-master-37910252-vmss000000"
... skipping 38 lines ...
I0922 04:20:03.701685       1 utils.go:79] GRPC request: {}
I0922 04:20:03.701751       1 utils.go:85] GRPC response: {"name":"disk.csi.azure.com","vendor_version":"latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad"}
I0922 04:20:03.793031       1 shared_state.go:413]  "msg"="Adding pod csi-azuredisk-controller-6956ccfdfb-5jfsx to shared map with keyName kube-system/csi-azuredisk-controller-6956ccfdfb-5jfsx." 
I0922 04:20:03.800510       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:20:03.801589       1 utils.go:79] GRPC request: {}
I0922 04:20:03.801749       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 04:20:03.803472       1 shared_state.go:419]  "msg"="Pod spec of pod csi-azuredisk-controller-6956ccfdfb-5jfsx is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-czmgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=15s --leader-election --leader-election-namespace=kube-system --worker-threads=40 --extra-create-metadata=true --strict-topology=true] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-czmgf ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000fe79c8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-master-37910252-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000fe79d0} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000fe79d8}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000fe79e0 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000fe79e4 PreemptionPolicy:0xc000ff6810 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-czmgf VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0922 04:20:03.803971       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.804207       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.804374       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.805221       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6956ccfdfb-5jfsx: Skipping Volume {kube-api-access-czmgf {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:20:03.805273       1 shared_state.go:483]  "msg"="Storing pod csi-azuredisk-controller-6956ccfdfb-5jfsx and claim [] to podToClaimsMap map." 
I0922 04:20:03.805439       1 utils.go:78] GRPC call: /csi.v1.Identity/GetPluginInfo
... skipping 297 lines ...
I0922 04:21:42.430410       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0922 04:21:42.430443       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-b2afb6b1-3270-4daa-b997-87a148097c19","parameters":{"csi.storage.k8s.io/pv/name":"pvc-b2afb6b1-3270-4daa-b997-87a148097c19","csi.storage.k8s.io/pvc/name":"pvc-jgfgf","csi.storage.k8s.io/pvc/namespace":"azuredisk-824","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0922 04:21:42.431193       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:21:42.445302       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 13 milliseconds
I0922 04:21:42.455750       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:21:42.455811       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-b2afb6b1-3270-4daa-b997-87a148097c19)
I0922 04:21:42.457244       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:21:42.522650       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b2afb6b1-3270-4daa-b997-87a148097c19/status 200 OK in 64 milliseconds
I0922 04:21:42.523184       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=65470001
I0922 04:21:42.523258       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:21:42.525768       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:21:42.574534       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:21:42.574955       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.051131665 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:21:42.574990       1 cloudprovisioner.go:246] begin to create disk(pvc-b2afb6b1-3270-4daa-b997-87a148097c19) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0922 04:21:42.593420       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018386415 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:21:42.593477       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-b2afb6b1-3270-4daa-b997-87a148097c19 StorageAccountType:StandardSSD_LRS Size:10
I0922 04:21:44.072168       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 04:21:44.076169       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:21:44.076394       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:21:44.969589       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.376025395 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:21:44.990584       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020923127 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:21:44.990626       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-b2afb6b1-3270-4daa-b997-87a148097c19 StorageAccountType:StandardSSD_LRS Size:10
I0922 04:21:44.990890       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-b2afb6b1-3270-4daa-b997-87a148097c19) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-b2afb6b1-3270-4daa-b997-87a148097c19 kubernetes.io-created-for-pvc-name:pvc-jgfgf kubernetes.io-created-for-pvc-namespace:azuredisk-824]) successfully" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:21:44.991084       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=2467538051
I0922 04:21:45.005030       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:21:45.005402       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=2549302274
I0922 04:21:45.005784       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=2574584795
I0922 04:21:45.005978       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.5748467059999998 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19" result_code="succeeded"
I0922 04:21:45.006007       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b2afb6b1-3270-4daa-b997-87a148097c19","csi.storage.k8s.io/pvc/name":"pvc-jgfgf","csi.storage.k8s.io/pvc/namespace":"azuredisk-824","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19"}}
I0922 04:21:45.008999       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b2afb6b1-3270-4daa-b997-87a148097c19/status 200 OK in 17 milliseconds
I0922 04:21:45.010059       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=18907438
I0922 04:21:45.010958       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=2486756402
I0922 04:21:45.011164       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=2553350653
I0922 04:21:45.493468       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0922 04:21:45.493505       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000003","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b2afb6b1-3270-4daa-b997-87a148097c19","csi.storage.k8s.io/pvc/name":"pvc-jgfgf","csi.storage.k8s.io/pvc/namespace":"azuredisk-824","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19"}
I0922 04:21:45.494226       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=3791k8s00000003,disk.csi.azure.com/volume-name!=pvc-b2afb6b1-3270-4daa-b997-87a148097c19." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:21:45.504076       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 9 milliseconds
I0922 04:21:45.504701       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b2afb6b1-3270-4daa-b997-87a148097c19-3791k8s00000003-attachment)
I0922 04:21:45.510516       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:21:45.512292       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b2afb6b1-3270-4daa-b997-87a148097c19-3791k8s00000003-attachment/status 200 OK in 10 milliseconds
I0922 04:21:45.513750       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=11755421
I0922 04:21:45.514121       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:21:45.536036       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02166736 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:21:45.536107       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:21:45.565811       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.029619912 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 9 lines ...
I0922 04:21:46.679251       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:21:46.703876       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.024555388 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:21:46.703912       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:21:46.703951       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:21:46.712149       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b2afb6b1-3270-4daa-b997-87a148097c19-3791k8s00000003-attachment/status 200 OK in 7 milliseconds
I0922 04:21:46.713828       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=9637727
I0922 04:21:46.715784       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:21:46.715852       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=1210886161
I0922 04:21:46.715901       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=1211222476
I0922 04:21:46.715940       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=1221890448
I0922 04:21:46.715997       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.222073157 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19" node="3791k8s00000003" result_code="succeeded"
I0922 04:21:46.716016       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 04:21:46.721207       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017219063 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:21:46.721232       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:21:46.721247       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19:AttachDiskOptions{diskName: "pvc-b2afb6b1-3270-4daa-b997-87a148097c19", lun: 0}]
I0922 04:21:46.721293       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19:AttachDiskOptions{diskName: "pvc-b2afb6b1-3270-4daa-b997-87a148097c19", lun: 0}])
I0922 04:21:48.090456       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 04:21:48.096454       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 04:21:48.096652       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:21:48.491417       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.770066845 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 04:21:48.491454       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19:AttachDiskOptions{diskName: "pvc-b2afb6b1-3270-4daa-b997-87a148097c19", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 04:21:50.106819       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0922 04:21:50.115651       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0922 04:21:50.116257       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:21:52.124625       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 04:21:52.131262       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 04:21:52.131464       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 153 lines ...
I0922 04:23:00.607485       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b2afb6b1-3270-4daa-b997-87a148097c19-3791k8s00000003-attachment/status 200 OK in 9 milliseconds
I0922 04:23:00.608409       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=11016988
I0922 04:23:00.618461       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-b2afb6b1-3270-4daa-b997-87a148097c19) scheduled in 5m0s." "disk.csi.azure.com/request-id"="3e4846d4-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:23:00.622876       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b2afb6b1-3270-4daa-b997-87a148097c19-3791k8s00000003-attachment 200 OK in 14 milliseconds
I0922 04:23:00.623285       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b2afb6b1-3270-4daa-b997-87a148097c19-3791k8s00000003-attachment)
I0922 04:23:00.625655       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b2afb6b1-3270-4daa-b997-87a148097c19-3791k8s00000003-attachment/status 200 OK in 7 milliseconds
I0922 04:23:00.626375       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:23:00.627867       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=9908139
I0922 04:23:00.627957       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:23:00.628228       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19 from node 3791k8s00000003" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:23:00.628279       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:23:00.651192       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022838411 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:23:00.651304       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
... skipping 12 lines ...
I0922 04:23:02.496518       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 04:23:02.502141       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 04:23:02.502338       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:23:02.790223       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:02.790243       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:02.790225       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:02.790320       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:23:03.379397       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:03.381574       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:03.383858       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:04.506081       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:23:04.509454       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:23:04.510129       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 10 lines ...
I0922 04:23:12.024950       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19:pvc-b2afb6b1-3270-4daa-b997-87a148097c19]) returned with <nil>
I0922 04:23:12.024989       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 3791k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19:pvc-b2afb6b1-3270-4daa-b997-87a148097c19]
I0922 04:23:12.025063       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-b2afb6b1-3270-4daa-b997-87a148097c19, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19) succeeded
I0922 04:23:12.025142       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19 from node 3791k8s00000003 successfully" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:23:12.025225       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=11397089167
I0922 04:23:12.039848       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b2afb6b1-3270-4daa-b997-87a148097c19-3791k8s00000003-attachment 200 OK in 14 milliseconds
I0922 04:23:12.041777       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:23:12.041876       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=11418284701
I0922 04:23:12.041946       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=11418669318
I0922 04:23:12.041990       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="3e450e1e-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=11444659569
I0922 04:23:12.042049       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=11.444814776 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19" node="3791k8s00000003" result_code="succeeded"
I0922 04:23:12.042143       1 utils.go:85] GRPC response: {}
I0922 04:23:12.046680       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="11813464-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=21364941
... skipping 18 lines ...
I0922 04:23:21.795084       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19"}
I0922 04:23:21.795256       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19)
I0922 04:23:21.808551       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b2afb6b1-3270-4daa-b997-87a148097c19/status 200 OK in 12 milliseconds
I0922 04:23:21.811273       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="4ae79f65-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=15865198
I0922 04:23:21.811302       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-b2afb6b1-3270-4daa-b997-87a148097c19)
I0922 04:23:21.821614       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b2afb6b1-3270-4daa-b997-87a148097c19 200 OK in 10 milliseconds
I0922 04:23:21.822960       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:23:21.830859       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b2afb6b1-3270-4daa-b997-87a148097c19/status 200 OK in 9 milliseconds
I0922 04:23:21.831401       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=10589765
I0922 04:23:21.831622       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:23:21.831895       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-b2afb6b1-3270-4daa-b997-87a148097c19)" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:23:21.832139       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-b2afb6b1-3270-4daa-b997-87a148097c19)" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:23:21.832388       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-b2afb6b1-3270-4daa-b997-87a148097c19." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19"
I0922 04:23:21.832705       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=811236
I0922 04:23:21.834193       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:23:21.861907       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028920971 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:23:22.602290       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:23:22.606852       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:23:22.607423       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:23:24.611224       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:23:24.620505       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
... skipping 3 lines ...
I0922 04:23:26.630702       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:23:27.030094       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.168109905 request="disks_delete" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:23:27.030544       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19
I0922 04:23:27.030828       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=5197827012
I0922 04:23:27.054605       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b2afb6b1-3270-4daa-b997-87a148097c19 200 OK in 23 milliseconds
I0922 04:23:27.055042       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=24029550
I0922 04:23:27.055472       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:23:27.055628       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="0fadd7c5-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=5223233222
I0922 04:23:27.055796       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="4ae79f65-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=5231630191
I0922 04:23:27.056029       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "disk.csi.azure.com/request-id"="4ae79f65-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=5260608165
I0922 04:23:27.056052       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19) returned with <nil>
I0922 04:23:27.056162       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.260813774 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19" result_code="succeeded"
I0922 04:23:27.056190       1 utils.go:85] GRPC response: {}
... skipping 15 lines ...
I0922 04:23:32.461072       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9","parameters":{"csi.storage.k8s.io/pv/name":"pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-znc2l-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8236","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0922 04:23:32.461638       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:23:32.497246       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 34 milliseconds
I0922 04:23:32.497793       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:23:32.497926       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9)
I0922 04:23:32.513612       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9/status 200 OK in 17 milliseconds
I0922 04:23:32.515125       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:23:32.517550       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=22211469
I0922 04:23:32.517594       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:23:32.544343       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:23:32.544680       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.026823571 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:23:32.544725       1 cloudprovisioner.go:246] begin to create disk(pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0922 04:23:32.582603       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.037786849 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:23:32.582650       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9 StorageAccountType:StandardSSD_LRS Size:10
I0922 04:23:32.678805       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 04:23:32.683276       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:23:32.683619       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:23:32.790334       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:32.790612       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:32.790628       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:32.790951       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:23:33.379691       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:33.381870       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:33.383987       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:23:34.688041       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:23:34.691662       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:23:34.691941       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 3 lines ...
I0922 04:23:34.849704       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9 kubernetes.io-created-for-pvc-name:test-volume-1-azuredisk-volume-tester-znc2l-0 kubernetes.io-created-for-pvc-namespace:azuredisk-8236]) successfully" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:23:34.849818       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=2332036601
I0922 04:23:34.867063       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9/status 200 OK in 16 milliseconds
I0922 04:23:34.868678       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=18676716
I0922 04:23:34.868933       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=2351263741
I0922 04:23:34.869188       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=2373867728
I0922 04:23:34.874645       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:23:34.874971       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=2376720152
I0922 04:23:34.875238       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=2413697867
I0922 04:23:34.875453       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.413938277 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" result_code="succeeded"
I0922 04:23:34.875686       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-znc2l-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8236","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"}}
I0922 04:23:35.559041       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0922 04:23:35.559070       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-znc2l-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8236","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"}
I0922 04:23:35.559845       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=3791k8s00000001,disk.csi.azure.com/volume-name!=pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:23:35.565525       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 5 milliseconds
I0922 04:23:35.568026       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment)
I0922 04:23:35.579351       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:23:35.580478       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment/status 200 OK in 10 milliseconds
I0922 04:23:35.581581       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=12302637
I0922 04:23:35.581779       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:23:35.603621       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02155184 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:23:35.603721       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:23:35.620653       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016806834 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 2 lines ...
I0922 04:23:35.621005       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9\" to node \"3791k8s00000001\"." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:23:36.621765       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-hpczfw5p|3791k8s00000001" delay="1s"
I0922 04:23:36.621866       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:23:36.641358       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019434349 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:23:36.641386       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:23:36.641432       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:23:36.653286       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:23:36.653669       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=1085462484
I0922 04:23:36.654195       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=1086129513
I0922 04:23:36.654474       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=1094720488
I0922 04:23:36.654592       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.094834093 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" node="3791k8s00000001" result_code="succeeded"
I0922 04:23:36.654695       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 04:23:36.660918       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019448749 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 3 lines ...
I0922 04:23:36.663365       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment/status 200 OK in 20 milliseconds
I0922 04:23:36.664312       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=22619887
I0922 04:23:36.695598       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:23:36.702597       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 04:23:36.702782       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:23:36.813361       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.151826528 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 04:23:36.813428       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9:AttachDiskOptions{diskName: "pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 04:23:38.709707       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 04:23:38.713226       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:23:38.713733       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:23:40.719700       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 04:23:40.724312       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 04:23:40.724858       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 144 lines ...
I0922 04:24:44.293193       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9) scheduled in 5m0s." "disk.csi.azure.com/request-id"="7c13c8df-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:24:44.300602       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment/status 200 OK in 8 milliseconds
I0922 04:24:44.301151       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=8880790
I0922 04:24:44.301202       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:24:44.301505       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9 from node 3791k8s00000001" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:24:44.301535       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:24:44.303428       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:24:44.328967       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.0273255 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:24:44.329009       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:24:44.329038       1 azure_vmss.go:204] Couldn't find VMSS VM with nodeName 3791k8s00000001, refreshing the cache(vmss: 3791k8s00, rg: kubetest-hpczfw5p)
I0922 04:24:44.415139       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.086050279 request="vmssvm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:24:45.065120       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:24:45.069071       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
... skipping 29 lines ...
I0922 04:25:00.656365       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=15.221907724 request="vmssvm_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="detach_disk" error_code=""
I0922 04:25:00.656405       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9:pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9]) returned with <nil>
I0922 04:25:00.656445       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 3791k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9:pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9]
I0922 04:25:00.656495       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9) succeeded
I0922 04:25:00.656542       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9 from node 3791k8s00000001 successfully" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:25:00.656588       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=16355173792
I0922 04:25:00.668927       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:25:00.669134       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=16376048013
I0922 04:25:00.669289       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="531be5c0-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=16376419229
I0922 04:25:00.670045       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7c0ee53a-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=16408888655
I0922 04:25:00.670296       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.409296573 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" node="3791k8s00000001" result_code="succeeded"
I0922 04:25:00.670475       1 utils.go:85] GRPC response: {}
I0922 04:25:00.675660       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment 200 OK in 18 milliseconds
... skipping 407 lines ...
I0922 04:28:29.061828       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 9 milliseconds
I0922 04:28:29.064530       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment)
I0922 04:28:29.065718       1 replica.go:157]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="7c13c8df-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileReplica).triggerGarbageCollection" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=224772544282
I0922 04:28:29.072832       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment/status 200 OK in 7 milliseconds
I0922 04:28:29.075146       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=9823851
I0922 04:28:29.075293       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:28:29.077279       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:28:29.148349       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.072720436 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:28:29.148436       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:28:29.169751       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021268976 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:28:29.169783       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:28:29.169812       1 azure_vmss.go:204] Couldn't find VMSS VM with nodeName 3791k8s00000001, refreshing the cache(vmss: 3791k8s00, rg: kubetest-hpczfw5p)
I0922 04:28:29.296981       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.127124731 request="vmssvm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 4 lines ...
I0922 04:28:30.238389       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:28:30.298053       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-hpczfw5p|3791k8s00000001" delay="1s"
I0922 04:28:30.298171       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:28:30.318392       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020179625 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:28:30.318477       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:28:30.318667       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:28:30.332377       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:28:30.332475       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=1267388136
I0922 04:28:30.332547       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=1268017965
I0922 04:28:30.332604       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=1281090365
I0922 04:28:30.332655       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.281269873 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" node="3791k8s00000001" result_code="succeeded"
I0922 04:28:30.332673       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 04:28:30.335628       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment/status 200 OK in 16 milliseconds
... skipping 3 lines ...
I0922 04:28:30.340112       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9:AttachDiskOptions{diskName: "pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9", lun: 0}])
I0922 04:28:30.341145       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=22318624
I0922 04:28:31.043765       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:28:31.043804       1 utils.go:79] GRPC request: {}
I0922 04:28:31.043865       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 04:28:31.084384       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.744219038 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 04:28:31.084424       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9:AttachDiskOptions{diskName: "pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 04:28:32.242917       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:28:32.256845       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 13 milliseconds
I0922 04:28:32.259094       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:28:32.799794       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:28:32.799917       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:28:32.800233       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 107 lines ...
I0922 04:29:10.579197       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment)
I0922 04:29:10.586423       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment/status 200 OK in 8 milliseconds
I0922 04:29:10.587055       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=9903949
I0922 04:29:10.587106       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:29:10.587402       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9 from node 3791k8s00000001" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:29:10.587461       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:29:10.589687       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:10.609473       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021838292 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:10.609511       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:29:10.609566       1 azure_vmss.go:204] Couldn't find VMSS VM with nodeName 3791k8s00000001, refreshing the cache(vmss: 3791k8s00, rg: kubetest-hpczfw5p)
I0922 04:29:10.709344       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.099612722 request="vmssvm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:11.710261       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-hpczfw5p|3791k8s00000001" delay="1s"
I0922 04:29:11.710325       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
... skipping 32 lines ...
I0922 04:29:27.025897       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9:pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9]) returned with <nil>
I0922 04:29:27.025937       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 3791k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9:pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9]
I0922 04:29:27.025979       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9) succeeded
I0922 04:29:27.026036       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9 from node 3791k8s00000001 successfully" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:29:27.026078       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=16438727845
I0922 04:29:27.037884       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9-3791k8s00000001-attachment 200 OK in 11 milliseconds
I0922 04:29:27.040403       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:29:27.040593       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=16461184366
I0922 04:29:27.040794       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=16461484880
I0922 04:29:27.040880       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="1ac8d612-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=16481121772
I0922 04:29:27.040939       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.481287779 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" node="3791k8s00000001" result_code="succeeded"
I0922 04:29:27.040990       1 utils.go:85] GRPC response: {}
I0922 04:29:27.042729       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="020b2f70-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=16606656
... skipping 23 lines ...
I0922 04:29:35.448616       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0922 04:29:35.448645       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"}
I0922 04:29:35.448785       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9)
I0922 04:29:35.462282       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9/status 200 OK in 13 milliseconds
I0922 04:29:35.462759       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="299e9f59-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=13833113
I0922 04:29:35.462780       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9)
I0922 04:29:35.464867       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:35.476023       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9 200 OK in 13 milliseconds
I0922 04:29:35.476495       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:35.486961       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9/status 200 OK in 11 milliseconds
I0922 04:29:35.488226       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=13550701
I0922 04:29:35.488522       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:29:35.488781       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9)" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:29:35.489003       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9)" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:29:35.489210       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9"
I0922 04:29:35.489403       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:35.489601       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=669030
I0922 04:29:35.517364       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.027538321 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:36.569091       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:29:36.573690       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:29:36.573880       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:29:38.577952       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
... skipping 3 lines ...
I0922 04:29:40.590585       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:29:40.590880       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:29:40.665829       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.148409289 request="disks_delete" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:40.665877       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9
I0922 04:29:40.666026       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=5176109917
I0922 04:29:40.679808       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9 200 OK in 13 milliseconds
I0922 04:29:40.681220       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:29:40.683193       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=17034880
I0922 04:29:40.683599       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=5194843174
I0922 04:29:40.683804       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="514326ff-3a2e-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=5209096206
I0922 04:29:40.683287       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="299e9f59-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=5206109473
I0922 04:29:40.684062       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "disk.csi.azure.com/request-id"="299e9f59-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9" "latency"=5235165462
I0922 04:29:40.684183       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9) returned with <nil>
... skipping 33 lines ...
I0922 04:29:50.858893       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 14 milliseconds
I0922 04:29:50.863392       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 43 milliseconds
I0922 04:29:50.863654       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:29:50.863671       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0)
I0922 04:29:50.863966       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:29:50.863983       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-2c697a4f-73c6-48d3-983c-75533c4d6552)
I0922 04:29:50.867122       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:50.874329       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:50.877714       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de/status 200 OK in 20 milliseconds
I0922 04:29:50.878368       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:50.878752       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=21691788
I0922 04:29:50.879023       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"
I0922 04:29:50.885778       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552/status 200 OK in 12 milliseconds
I0922 04:29:50.886404       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0/status 200 OK in 17 milliseconds
I0922 04:29:50.887006       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=18341135
I0922 04:29:50.887200       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:29:50.888137       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=14859877
I0922 04:29:50.888354       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:29:50.891112       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:50.894939       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:50.899724       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:29:50.900048       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02063444 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:29:50.900249       1 cloudprovisioner.go:246] begin to create disk(pvc-91bdc1e7-22db-44e4-a356-51048dfc34de) account type(Premium_LRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0922 04:29:50.916583       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:29:50.916704       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.029106326 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:29:50.916731       1 cloudprovisioner.go:246] begin to create disk(pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0) account type(Standard_LRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0922 04:29:50.924505       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:29:50.924623       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.035801431 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:29:50.924713       1 cloudprovisioner.go:246] begin to create disk(pvc-2c697a4f-73c6-48d3-983c-75533c4d6552) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0922 04:29:50.954396       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.029443641 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:50.954438       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-2c697a4f-73c6-48d3-983c-75533c4d6552 StorageAccountType:StandardSSD_LRS Size:10
I0922 04:29:50.967466       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.0504937 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:50.967507       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0 StorageAccountType:Standard_LRS Size:10
... skipping 4 lines ...
I0922 04:29:52.650410       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:29:53.190846       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.236306784 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:53.215241       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.024320108 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:53.215301       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-2c697a4f-73c6-48d3-983c-75533c4d6552 StorageAccountType:StandardSSD_LRS Size:10
I0922 04:29:53.215366       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-2c697a4f-73c6-48d3-983c-75533c4d6552) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-2c697a4f-73c6-48d3-983c-75533c4d6552 kubernetes.io-created-for-pvc-name:pvc-kd9wj kubernetes.io-created-for-pvc-namespace:azuredisk-3843]) successfully" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:29:53.215408       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=2326697602
I0922 04:29:53.244781       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:29:53.244912       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=2380854668
I0922 04:29:53.244985       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=2400639671
I0922 04:29:53.245063       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.400707274 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" result_code="succeeded"
I0922 04:29:53.245085       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2c697a4f-73c6-48d3-983c-75533c4d6552","csi.storage.k8s.io/pvc/name":"pvc-kd9wj","csi.storage.k8s.io/pvc/namespace":"azuredisk-3843","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"}}
I0922 04:29:53.246786       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.279236439 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:53.265731       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552/status 200 OK in 50 milliseconds
... skipping 7 lines ...
I0922 04:29:53.273061       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.279974873 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:53.296262       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023148155 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:53.296300       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-91bdc1e7-22db-44e4-a356-51048dfc34de StorageAccountType:Premium_LRS Size:10
I0922 04:29:53.296372       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-91bdc1e7-22db-44e4-a356-51048dfc34de) account type(Premium_LRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-91bdc1e7-22db-44e4-a356-51048dfc34de kubernetes.io-created-for-pvc-name:pvc-bjhnp kubernetes.io-created-for-pvc-namespace:azuredisk-3843]) successfully" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"
I0922 04:29:53.296432       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=2417119721
I0922 04:29:53.299470       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0/status 200 OK in 26 milliseconds
I0922 04:29:53.301929       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:29:53.302026       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=2438262285
I0922 04:29:53.302089       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=2482701509
I0922 04:29:53.302127       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.482787013 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" result_code="succeeded"
I0922 04:29:53.302144       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0","csi.storage.k8s.io/pvc/name":"pvc-sr5ps","csi.storage.k8s.io/pvc/namespace":"azuredisk-3843","requestedsizegib":"10","skuname":"Standard_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"}}
I0922 04:29:53.305982       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=33655333
I0922 04:29:53.306071       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=2418547586
I0922 04:29:53.306105       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=2437461647
I0922 04:29:53.342303       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de/status 200 OK in 45 milliseconds
I0922 04:29:53.344313       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:29:53.344525       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=2489059499
I0922 04:29:53.344765       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=2530985709
I0922 04:29:53.344997       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.531401228 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" result_code="succeeded"
I0922 04:29:53.345137       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-91bdc1e7-22db-44e4-a356-51048dfc34de","csi.storage.k8s.io/pvc/name":"pvc-bjhnp","csi.storage.k8s.io/pvc/namespace":"azuredisk-3843","requestedsizegib":"10","skuname":"Premium_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"}}
I0922 04:29:53.356770       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=60227144
I0922 04:29:53.357010       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=2477751084
... skipping 10 lines ...
I0922 04:29:53.908993       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 9 milliseconds
I0922 04:29:53.913381       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0-3791k8s00000001-attachment)
I0922 04:29:53.914559       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 18 milliseconds
I0922 04:29:53.916368       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 19 milliseconds
I0922 04:29:53.917117       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-91bdc1e7-22db-44e4-a356-51048dfc34de-3791k8s00000001-attachment)
I0922 04:29:53.917879       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2c697a4f-73c6-48d3-983c-75533c4d6552-3791k8s00000001-attachment)
I0922 04:29:53.926408       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:53.926578       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:53.937267       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0-3791k8s00000001-attachment/status 200 OK in 25 milliseconds
I0922 04:29:53.938308       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=26774219
I0922 04:29:53.938404       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:29:53.942651       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de-3791k8s00000001-attachment/status 200 OK in 14 milliseconds
I0922 04:29:53.944557       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:53.949740       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=22095006
I0922 04:29:53.950004       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"
I0922 04:29:53.950808       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552-3791k8s00000001-attachment/status 200 OK in 23 milliseconds
I0922 04:29:53.951421       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=24048296
I0922 04:29:53.952216       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:29:53.954692       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:53.956207       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:29:53.975561       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022991948 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:53.975771       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:29:53.976260       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02589368 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:53.984723       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.045893291 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:53.993578       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017707307 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:29:53.993608       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
... skipping 20 lines ...
I0922 04:29:55.151929       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:29:55.152145       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:29:55.169594       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de-3791k8s00000001-attachment/status 200 OK in 17 milliseconds
I0922 04:29:55.169876       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552-3791k8s00000001-attachment/status 200 OK in 16 milliseconds
I0922 04:29:55.172105       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=19293679
I0922 04:29:55.172493       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=19760200
I0922 04:29:55.172999       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:29:55.173243       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:29:55.173637       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=1256132928
I0922 04:29:55.173992       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=1255868115
I0922 04:29:55.174215       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=1256350138
I0922 04:29:55.174446       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=1279414589
I0922 04:29:55.174655       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.280346931 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" node="3791k8s00000001" result_code="succeeded"
I0922 04:29:55.174839       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
... skipping 3 lines ...
I0922 04:29:55.177836       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552:AttachDiskOptions{diskName: "pvc-2c697a4f-73c6-48d3-983c-75533c4d6552", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0:AttachDiskOptions{diskName: "pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de:AttachDiskOptions{diskName: "pvc-91bdc1e7-22db-44e4-a356-51048dfc34de", lun: 2}]
I0922 04:29:55.177914       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552:AttachDiskOptions{diskName: "pvc-2c697a4f-73c6-48d3-983c-75533c4d6552", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0:AttachDiskOptions{diskName: "pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de:AttachDiskOptions{diskName: "pvc-91bdc1e7-22db-44e4-a356-51048dfc34de", lun: 2}])
I0922 04:29:55.178305       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=1280628544
I0922 04:29:55.178394       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.281911503 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" node="3791k8s00000001" result_code="succeeded"
I0922 04:29:55.178454       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0922 04:29:55.193877       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0-3791k8s00000001-attachment/status 200 OK in 40 milliseconds
I0922 04:29:55.203419       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:29:55.203519       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=1289898266
I0922 04:29:55.203827       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=1290464492
I0922 04:29:55.204075       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=1305276367
I0922 04:29:55.204128       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.305724487 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" node="3791k8s00000001" result_code="succeeded"
I0922 04:29:55.204148       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 04:29:55.213944       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 9 lines ...
I0922 04:29:55.216844       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="3566e603-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=42302
I0922 04:29:55.216887       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="3566e603-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=528624
I0922 04:29:55.217058       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.000719033 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" node="3791k8s00000001" result_code="succeeded"
I0922 04:29:55.217231       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0922 04:29:55.220820       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=68569924
I0922 04:29:55.424972       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.247003953 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 04:29:55.425020       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552:AttachDiskOptions{diskName: "pvc-2c697a4f-73c6-48d3-983c-75533c4d6552", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0:AttachDiskOptions{diskName: "pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de:AttachDiskOptions{diskName: "pvc-91bdc1e7-22db-44e4-a356-51048dfc34de", lun: 2}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 04:29:56.666342       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:29:56.670041       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:29:56.670377       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:29:58.673440       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:29:58.676961       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:29:58.677221       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 162 lines ...
I0922 04:30:52.108385       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0-3791k8s00000001-attachment) detachment" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:30:52.119632       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0-3791k8s00000001-attachment/status 200 OK in 10 milliseconds
I0922 04:30:52.120088       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=11622011
I0922 04:30:52.142005       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0-3791k8s00000001-attachment 200 OK in 21 milliseconds
I0922 04:30:52.144276       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0) scheduled in 5m0s." "disk.csi.azure.com/request-id"="57556539-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:30:52.153501       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0-3791k8s00000001-attachment)
I0922 04:30:52.154727       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:30:52.160834       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:30:52.164915       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0-3791k8s00000001-attachment/status 200 OK in 21 milliseconds
I0922 04:30:52.166319       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=22898407
I0922 04:30:52.166547       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:30:52.166907       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0 from node 3791k8s00000001" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:30:52.167103       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:30:52.173077       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 21 lines ...
I0922 04:30:52.243417       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:30:52.243520       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552 from node 3791k8s00000001" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:30:52.243539       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:30:52.244212       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=17275760
I0922 04:30:52.244247       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"
I0922 04:30:52.244345       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de from node 3791k8s00000001" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"
I0922 04:30:52.249307       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:30:52.249341       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:30:52.262440       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018868529 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:30:52.262469       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:30:52.262560       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:30:52.278312       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.086277194 request="vmssvm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:30:52.281087       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018389708 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:30:52.281340       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
... skipping 22 lines ...
I0922 04:31:01.044224       1 utils.go:79] GRPC request: {}
I0922 04:31:01.044362       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 04:31:01.165115       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 04:31:01.180595       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 14 milliseconds
I0922 04:31:01.181241       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:31:02.803141       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:31:02.803278       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:02.803315       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:02.803334       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:02.804283       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:31:02.804290       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:31:03.187806       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 04:31:03.200371       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 12 milliseconds
I0922 04:31:03.201031       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:31:03.393621       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 29 lines ...
I0922 04:31:09.783284       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=17538991527
I0922 04:31:09.783321       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=17556406893
I0922 04:31:09.786531       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552-3791k8s00000001-attachment 200 OK in 25 milliseconds
I0922 04:31:09.789062       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=28225202
I0922 04:31:09.789112       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=17545648349
I0922 04:31:09.789145       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=17559891276
I0922 04:31:09.791208       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:09.791285       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=17637504662
I0922 04:31:09.791375       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349dd2c3-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=17637862878
I0922 04:31:09.791416       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="574fed50-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=17683068965
I0922 04:31:09.791506       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.683388278 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" node="3791k8s00000001" result_code="succeeded"
I0922 04:31:09.791523       1 utils.go:85] GRPC response: {}
I0922 04:31:09.793631       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:09.793718       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=17565099850
I0922 04:31:09.793797       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d8661-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=17565301158
I0922 04:31:09.793861       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="575b3b6c-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=17611399284
I0922 04:31:09.793928       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.611516188 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" node="3791k8s00000001" result_code="succeeded"
I0922 04:31:09.793948       1 utils.go:85] GRPC response: {}
I0922 04:31:09.800155       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:09.800256       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=17564795758
I0922 04:31:09.800303       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="349d470f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=17565615293
I0922 04:31:09.800339       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="575a27b1-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=17624961602
I0922 04:31:09.800375       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.625059606 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" node="3791k8s00000001" result_code="succeeded"
I0922 04:31:09.800391       1 utils.go:85] GRPC response: {}
I0922 04:31:09.825457       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 29 lines ...
I0922 04:31:21.747767       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0922 04:31:21.747797       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"}
I0922 04:31:21.747903       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552)
I0922 04:31:21.759710       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552/status 200 OK in 11 milliseconds
I0922 04:31:21.760882       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="68fa94fd-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=12798774
I0922 04:31:21.761125       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-2c697a4f-73c6-48d3-983c-75533c4d6552)
I0922 04:31:21.763309       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:21.771875       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552 200 OK in 10 milliseconds
I0922 04:31:21.773740       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:21.780855       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552/status 200 OK in 5 milliseconds
I0922 04:31:21.783221       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=8497581
I0922 04:31:21.783352       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:31:21.783668       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-2c697a4f-73c6-48d3-983c-75533c4d6552)" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:31:21.783912       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-2c697a4f-73c6-48d3-983c-75533c4d6552)" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:31:21.784132       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-2c697a4f-73c6-48d3-983c-75533c4d6552." "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552"
I0922 04:31:21.784338       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=674830
I0922 04:31:21.786980       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:21.809695       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.025123427 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:23.332099       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 04:31:23.355591       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 22 milliseconds
I0922 04:31:23.356569       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:31:25.364842       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 04:31:25.368576       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:31:25.368862       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:31:26.964955       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.155153515 request="disks_delete" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:26.964989       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552
I0922 04:31:26.965053       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=5180465050
I0922 04:31:26.991962       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552 200 OK in 26 milliseconds
I0922 04:31:26.994134       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:26.994531       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="68fa94fd-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=5221742390
I0922 04:31:26.994630       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="68fa94fd-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=5246605804
I0922 04:31:26.994649       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552) returned with <nil>
I0922 04:31:26.994725       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.246772712 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" result_code="succeeded"
I0922 04:31:26.994747       1 utils.go:85] GRPC response: {}
I0922 04:31:26.995651       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "disk.csi.azure.com/request-id"="32cbc83d-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2c697a4f-73c6-48d3-983c-75533c4d6552" "latency"=30539157
... skipping 22 lines ...
I0922 04:31:31.552673       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"}
I0922 04:31:31.552995       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de)
I0922 04:31:31.579410       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de/status 200 OK in 25 milliseconds
I0922 04:31:31.585973       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="6ed2c43b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=32491545
I0922 04:31:31.586202       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-91bdc1e7-22db-44e4-a356-51048dfc34de)
I0922 04:31:31.603812       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de 200 OK in 17 milliseconds
I0922 04:31:31.611895       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:31.624773       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de/status 200 OK in 16 milliseconds
I0922 04:31:31.628545       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=20455309
I0922 04:31:31.628826       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"
I0922 04:31:31.629153       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-91bdc1e7-22db-44e4-a356-51048dfc34de)" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"
I0922 04:31:31.629394       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-91bdc1e7-22db-44e4-a356-51048dfc34de)" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"
I0922 04:31:31.629698       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-91bdc1e7-22db-44e4-a356-51048dfc34de." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de"
I0922 04:31:31.630135       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=830537
I0922 04:31:31.641911       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:31.662210       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.031768212 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:32.803733       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:31:32.804808       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:31:32.804813       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:31:32.805193       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:33.393929       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:31:33.394092       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:31:33.395154       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:31:33.439460       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 24 milliseconds
I0922 04:31:33.447294       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 04:31:33.447630       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 4 lines ...
I0922 04:31:36.818879       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de
I0922 04:31:36.818944       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=5188501323
I0922 04:31:36.832641       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de 200 OK in 13 milliseconds
I0922 04:31:36.834438       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=15119761
I0922 04:31:36.834959       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=5205808380
I0922 04:31:36.835202       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="32c71b19-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=5227171329
I0922 04:31:36.834803       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:36.835388       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="6ed2c43b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=5228639595
I0922 04:31:36.835463       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "disk.csi.azure.com/request-id"="6ed2c43b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" "latency"=5282023268
I0922 04:31:36.835479       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de) returned with <nil>
I0922 04:31:36.835563       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.282500289 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-91bdc1e7-22db-44e4-a356-51048dfc34de" result_code="succeeded"
I0922 04:31:36.835594       1 utils.go:85] GRPC response: {}
I0922 04:31:37.465987       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
... skipping 13 lines ...
I0922 04:31:41.934280       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0922 04:31:41.934411       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"}
I0922 04:31:41.934667       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0)
I0922 04:31:41.949130       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0/status 200 OK in 13 milliseconds
I0922 04:31:41.949909       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="7502da6b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=15011375
I0922 04:31:41.950111       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0)
I0922 04:31:41.959734       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:41.961470       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0 200 OK in 10 milliseconds
I0922 04:31:41.975533       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0/status 200 OK in 12 milliseconds
I0922 04:31:41.975970       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:41.976349       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=13994930
I0922 04:31:41.976424       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:31:41.976629       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0)" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:31:41.976986       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0)" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:31:41.977095       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0"
I0922 04:31:41.977219       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=610627
... skipping 8 lines ...
I0922 04:31:47.142167       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0
I0922 04:31:47.142262       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=5164909782
I0922 04:31:47.193624       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0 200 OK in 50 milliseconds
I0922 04:31:47.194055       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=51725327
I0922 04:31:47.194102       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=5217514048
I0922 04:31:47.194135       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="32c7fc2f-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=5231801191
I0922 04:31:47.194732       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:47.194793       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="7502da6b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=5225868324
I0922 04:31:47.194841       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "disk.csi.azure.com/request-id"="7502da6b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" "latency"=5259953657
I0922 04:31:47.194857       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0) returned with <nil>
I0922 04:31:47.194900       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.2602066690000004 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7a4db4e1-9d7b-4376-ba48-43e192b575f0" result_code="succeeded"
I0922 04:31:47.194919       1 utils.go:85] GRPC response: {}
I0922 04:31:47.531550       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 17 lines ...
I0922 04:31:53.125495       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0922 04:31:53.125510       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-bcs4t-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","skuname":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0922 04:31:53.125751       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:31:53.135730       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0922 04:31:53.135760       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7","parameters":{"csi.storage.k8s.io/pv/name":"pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-jqbn4-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","skuname":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0922 04:31:53.136045       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:31:53.151875       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.156452       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7e5c622e-5895-4293-b7bf-415f682233a4/status 200 OK in 36 milliseconds
I0922 04:31:53.157107       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=37146956
I0922 04:31:53.157206       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:31:53.157484       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-7e5c622e-5895-4293-b7bf-415f682233a4) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:31:53.159385       1 shared_state.go:413]  "msg"="Adding pod azuredisk-volume-tester-jqbn4-0 to shared map with keyName azuredisk-8320/azuredisk-volume-tester-jqbn4-0." 
I0922 04:31:53.159885       1 shared_state.go:419]  "msg"="Pod spec of pod azuredisk-volume-tester-jqbn4-0 is: {Volumes:[{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-1-azuredisk-volume-tester-jqbn4-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-2 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-2-azuredisk-volume-tester-jqbn4-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-3 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-3-azuredisk-volume-tester-jqbn4-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-w28fp VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:volume-tester Image:mcr.microsoft.com/windows/servercore:ltsc2019 Command:[powershell.exe] Args:[-Command while (1) { Add-Content -Encoding Ascii C:\\mnt\\test-1\\data.txt $(Get-Date -Format u); sleep 3600 }] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:test-volume-1 ReadOnly:false MountPath:/mnt/test-1 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:test-volume-2 ReadOnly:false MountPath:/mnt/test-2 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:test-volume-3 ReadOnly:false MountPath:/mnt/test-3 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-w28fp ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc00108ab60 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:windows] ServiceAccountName:default DeprecatedServiceAccount:default AutomountServiceAccountToken:<nil> NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname:azuredisk-volume-tester-jqbn4-0 Subdomain: Affinity:nil SchedulerName:csi-azuredisk-scheduler-extender Tolerations:[{Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00108ab68} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00108ab70}] HostAliases:[] PriorityClassName: Priority:0xc00108ab78 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc00108ab7c PreemptionPolicy:0xc000f6d370 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-1-azuredisk-volume-tester-jqbn4-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-2 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-2-azuredisk-volume-tester-jqbn4-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-3 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-3-azuredisk-volume-tester-jqbn4-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-w28fp VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0922 04:31:53.161241       1 shared_state.go:457]  "msg"="Pod azuredisk-volume-tester-jqbn4-0: Skipping Volume {kube-api-access-w28fp {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:31:53.161289       1 shared_state.go:483]  "msg"="Storing pod azuredisk-volume-tester-jqbn4-0 and claim [] to podToClaimsMap map." 
I0922 04:31:53.166304       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 40 milliseconds
I0922 04:31:53.170820       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:31:53.171007       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e)
I0922 04:31:53.173806       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 48 milliseconds
I0922 04:31:53.176906       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:31:53.177016       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019429866 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:31:53.177043       1 cloudprovisioner.go:246] begin to create disk(pvc-7e5c622e-5895-4293-b7bf-415f682233a4) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone() maxShares(0)
I0922 04:31:53.188093       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.191727       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:31:53.191790       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39)
I0922 04:31:53.201162       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0922 04:31:53.201216       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4","parameters":{"csi.storage.k8s.io/pv/name":"pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-bcs4t-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","skuname":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0922 04:31:53.201696       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:31:53.202303       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 66 milliseconds
I0922 04:31:53.206797       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.029706824 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:53.206943       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-7e5c622e-5895-4293-b7bf-415f682233a4 StorageAccountType:StandardSSD_ZRS Size:10
I0922 04:31:53.211201       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:31:53.211233       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7)
I0922 04:31:53.214646       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.216025       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.236965       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e/status 200 OK in 67 milliseconds
I0922 04:31:53.247168       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=78027879
I0922 04:31:53.247222       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:31:53.247328       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:31:53.248209       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.254204       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39/status 200 OK in 40 milliseconds
I0922 04:31:53.260935       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7/status 200 OK in 50 milliseconds
I0922 04:31:53.261819       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=51454594
I0922 04:31:53.262129       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:31:53.262269       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:31:53.271287       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.272124       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=58189394
I0922 04:31:53.272170       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:31:53.272267       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:31:53.279725       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0922 04:31:53.279745       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4","parameters":{"csi.storage.k8s.io/pv/name":"pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-bcs4t-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","skuname":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0922 04:31:53.280121       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:31:53.312648       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:31:53.313086       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.05068846 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:31:53.313128       1 cloudprovisioner.go:246] begin to create disk(pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone() maxShares(0)
I0922 04:31:53.318765       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.320312       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:31:53.320418       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.072996655 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:31:53.320449       1 cloudprovisioner.go:246] begin to create disk(pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone() maxShares(0)
I0922 04:31:53.335602       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:31:53.335712       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.063344524 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:31:53.335739       1 cloudprovisioner.go:246] begin to create disk(pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone() maxShares(0)
I0922 04:31:53.384679       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.064170261 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:53.385151       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e StorageAccountType:StandardSSD_ZRS Size:10
I0922 04:31:53.385864       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.050080633 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:53.385890       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39 StorageAccountType:StandardSSD_ZRS Size:10
... skipping 3 lines ...
I0922 04:31:53.387925       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:31:53.388139       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4)
I0922 04:31:53.389283       1 shared_state.go:413]  "msg"="Adding pod azuredisk-volume-tester-bcs4t-0 to shared map with keyName azuredisk-8320/azuredisk-volume-tester-bcs4t-0." 
I0922 04:31:53.391110       1 shared_state.go:419]  "msg"="Pod spec of pod azuredisk-volume-tester-bcs4t-0 is: {Volumes:[{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-1-azuredisk-volume-tester-bcs4t-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-2 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-2-azuredisk-volume-tester-bcs4t-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-3 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-3-azuredisk-volume-tester-bcs4t-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-b87zx VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:volume-tester Image:mcr.microsoft.com/windows/servercore:ltsc2019 Command:[powershell.exe] Args:[-Command while (1) { Add-Content -Encoding Ascii C:\\mnt\\test-1\\data.txt $(Get-Date -Format u); sleep 3600 }] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:test-volume-1 ReadOnly:false MountPath:/mnt/test-1 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:test-volume-2 ReadOnly:false MountPath:/mnt/test-2 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:test-volume-3 ReadOnly:false MountPath:/mnt/test-3 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-b87zx ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc001109c90 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:windows] ServiceAccountName:default DeprecatedServiceAccount:default AutomountServiceAccountToken:<nil> NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname:azuredisk-volume-tester-bcs4t-0 Subdomain: Affinity:nil SchedulerName:csi-azuredisk-scheduler-extender Tolerations:[{Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001109c98} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001109ca0}] HostAliases:[] PriorityClassName: Priority:0xc001109ca8 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc001109cac PreemptionPolicy:0xc000434350 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-1-azuredisk-volume-tester-bcs4t-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-2 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-2-azuredisk-volume-tester-bcs4t-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-3 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-3-azuredisk-volume-tester-bcs4t-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-b87zx VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0922 04:31:53.391367       1 shared_state.go:457]  "msg"="Pod azuredisk-volume-tester-bcs4t-0: Skipping Volume {kube-api-access-b87zx {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:31:53.391392       1 shared_state.go:483]  "msg"="Storing pod azuredisk-volume-tester-bcs4t-0 and claim [] to podToClaimsMap map." 
I0922 04:31:53.391659       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.472469       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 192 milliseconds
I0922 04:31:53.477169       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:31:53.477574       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4)
I0922 04:31:53.515047       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4/status 200 OK in 124 milliseconds
I0922 04:31:53.516978       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.519402       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=130065699
I0922 04:31:53.519638       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:31:53.519973       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:31:53.529336       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4/status 200 OK in 59 milliseconds
I0922 04:31:53.530803       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:53.531568       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=61967763
I0922 04:31:53.531827       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:31:53.532123       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:31:53.536841       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:31:53.537138       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01684045 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:31:53.537356       1 cloudprovisioner.go:246] begin to create disk(pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone() maxShares(0)
I0922 04:31:53.552980       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:31:53.553304       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02085183 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:31:53.553488       1 cloudprovisioner.go:246] begin to create disk(pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone() maxShares(0)
I0922 04:31:53.577899       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0922 04:31:53.583643       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 04:31:53.584557       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:31:53.588141       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.034450437 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 6 lines ...
I0922 04:31:55.497654       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-7e5c622e-5895-4293-b7bf-415f682233a4) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-7e5c622e-5895-4293-b7bf-415f682233a4 kubernetes.io-created-for-pvc-name:test-volume-1-azuredisk-volume-tester-jqbn4-0 kubernetes.io-created-for-pvc-namespace:azuredisk-8320]) successfully" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:31:55.497877       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=2340343457
I0922 04:31:55.512933       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7e5c622e-5895-4293-b7bf-415f682233a4/status 200 OK in 14 milliseconds
I0922 04:31:55.514034       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=16051216
I0922 04:31:55.514100       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=2356774989
I0922 04:31:55.514138       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=2394239260
I0922 04:31:55.517257       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:55.517357       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=2395345409
I0922 04:31:55.517511       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=2440558625
I0922 04:31:55.517617       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.440683031 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4" result_code="succeeded"
I0922 04:31:55.517719       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7e5c622e-5895-4293-b7bf-415f682233a4","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-jqbn4-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4"}}
I0922 04:31:55.588965       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:31:55.592555       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
... skipping 5 lines ...
I0922 04:31:55.704128       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7 StorageAccountType:StandardSSD_ZRS Size:10
I0922 04:31:55.704193       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7 kubernetes.io-created-for-pvc-name:test-volume-3-azuredisk-volume-tester-jqbn4-0 kubernetes.io-created-for-pvc-namespace:azuredisk-8320]) successfully" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:31:55.704310       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=2441993289
I0922 04:31:55.704513       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e StorageAccountType:StandardSSD_ZRS Size:10
I0922 04:31:55.705085       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e kubernetes.io-created-for-pvc-name:test-volume-3-azuredisk-volume-tester-bcs4t-0 kubernetes.io-created-for-pvc-namespace:azuredisk-8320]) successfully" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:31:55.705284       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=2457821795
I0922 04:31:55.717109       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:55.717308       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=2505950541
I0922 04:31:55.717488       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=2581411606
I0922 04:31:55.717636       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.5816283159999998 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" result_code="succeeded"
I0922 04:31:55.717773       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-jqbn4-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"}}
I0922 04:31:55.723325       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7/status 200 OK in 17 milliseconds
I0922 04:31:55.724445       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=19979491
I0922 04:31:55.724727       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=2462477502
I0922 04:31:55.724998       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=2514534124
I0922 04:31:55.728182       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e/status 200 OK in 22 milliseconds
I0922 04:31:55.729600       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=24247481
I0922 04:31:55.729891       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=2482554198
I0922 04:31:55.730122       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=2560843189
I0922 04:31:55.732987       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:55.733355       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=2561222906
I0922 04:31:55.733637       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=2607859686
I0922 04:31:55.733945       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.608172599 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" result_code="succeeded"
I0922 04:31:55.734151       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-bcs4t-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"}}
I0922 04:31:55.753275       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.36733686 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:55.776368       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023018326 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:55.776407       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39 StorageAccountType:StandardSSD_ZRS Size:10
I0922 04:31:55.776620       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39 kubernetes.io-created-for-pvc-name:test-volume-2-azuredisk-volume-tester-jqbn4-0 kubernetes.io-created-for-pvc-namespace:azuredisk-8320]) successfully" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:31:55.776676       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=2504415773
I0922 04:31:55.793084       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.204698009 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:55.794530       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39/status 200 OK in 17 milliseconds
I0922 04:31:55.800475       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:55.800836       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=2608827329
I0922 04:31:55.801071       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=2676268836
I0922 04:31:55.801518       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.676607251 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" result_code="succeeded"
I0922 04:31:55.801656       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-jqbn4-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"}}
I0922 04:31:55.804638       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=27899344
I0922 04:31:55.805014       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=2532784837
I0922 04:31:55.805345       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=2591423953
I0922 04:31:55.816924       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023776661 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:55.817170       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4 StorageAccountType:StandardSSD_ZRS Size:10
I0922 04:31:55.817490       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4 kubernetes.io-created-for-pvc-name:test-volume-2-azuredisk-volume-tester-bcs4t-0 kubernetes.io-created-for-pvc-namespace:azuredisk-8320]) successfully" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:31:55.817777       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=2285625517
I0922 04:31:55.836934       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4/status 200 OK in 18 milliseconds
I0922 04:31:55.838004       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:55.838172       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=2360235844
I0922 04:31:55.838309       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=2558155869
I0922 04:31:55.838376       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.558309076 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" result_code="succeeded"
I0922 04:31:55.838397       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-bcs4t-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"}}
I0922 04:31:55.841542       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=23413344
I0922 04:31:55.842138       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=2309537483
I0922 04:31:55.842284       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=2372673298
I0922 04:31:55.978992       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.384421722 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:56.000782       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021682867 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:56.000822       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4 StorageAccountType:StandardSSD_ZRS Size:10
I0922 04:31:56.000917       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4 kubernetes.io-created-for-pvc-name:test-volume-1-azuredisk-volume-tester-bcs4t-0 kubernetes.io-created-for-pvc-namespace:azuredisk-8320]) successfully" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:31:56.001019       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=2481004829
I0922 04:31:56.019169       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:56.019590       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=2630446793
I0922 04:31:56.019917       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=2818189465
I0922 04:31:56.020260       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.818592383 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" result_code="succeeded"
I0922 04:31:56.020381       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-bcs4t-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"}}
I0922 04:31:56.028957       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4/status 200 OK in 26 milliseconds
I0922 04:31:56.029931       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=28844888
... skipping 15 lines ...
I0922 04:31:57.258379       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7-3791k8s00000001-attachment)
I0922 04:31:57.282694       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 48 milliseconds
I0922 04:31:57.282978       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7-3791k8s00000001-attachment/status 200 OK in 29 milliseconds
I0922 04:31:57.283807       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4-3791k8s00000000-attachment)
I0922 04:31:57.284516       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=30883880
I0922 04:31:57.284624       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:31:57.290617       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.295397       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.295576       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 39 milliseconds
I0922 04:31:57.297131       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4-3791k8s00000000-attachment)
I0922 04:31:57.297504       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 53 milliseconds
I0922 04:31:57.298054       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e-3791k8s00000000-attachment)
I0922 04:31:57.317571       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4-3791k8s00000000-attachment/status 200 OK in 27 milliseconds
I0922 04:31:57.317985       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.318424       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.033466095 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:57.318481       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:31:57.320032       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.320060       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.327131       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=37255065
I0922 04:31:57.327178       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:31:57.331768       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e-3791k8s00000000-attachment/status 200 OK in 26 milliseconds
I0922 04:31:57.334851       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.335670       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=30475861
I0922 04:31:57.335706       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:31:57.336833       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018302118 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:57.336855       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:31:57.345867       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4-3791k8s00000000-attachment/status 200 OK in 39 milliseconds
I0922 04:31:57.347757       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=41788167
I0922 04:31:57.347805       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:31:57.348573       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021281151 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:57.348621       1 azure_vmss_cache.go:332] Node 3791k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:31:57.349202       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.350664       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0922 04:31:57.350680       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7e5c622e-5895-4293-b7bf-415f682233a4","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-jqbn4-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4"}
I0922 04:31:57.350992       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=3791k8s00000001,disk.csi.azure.com/volume-name!=pvc-7e5c622e-5895-4293-b7bf-415f682233a4." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:31:57.358177       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022367 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:57.365104       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 13 milliseconds
I0922 04:31:57.369457       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7e5c622e-5895-4293-b7bf-415f682233a4-3791k8s00000001-attachment)
I0922 04:31:57.369695       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02103344 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:57.369712       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:31:57.369732       1 azure_vmss_cache.go:332] Node 3791k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:31:57.374871       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7e5c622e-5895-4293-b7bf-415f682233a4-3791k8s00000001-attachment/status 200 OK in 12 milliseconds
I0922 04:31:57.376416       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028503374 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:57.377597       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.378921       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=16280427
I0922 04:31:57.379048       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:31:57.402312       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022872122 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:57.410579       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.073684793 request="vmss_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:57.410844       1 azure_vmss.go:204] Couldn't find VMSS VM with nodeName 3791k8s00000001, refreshing the cache(vmss: 3791k8s00, rg: kubetest-hpczfw5p)
I0922 04:31:57.411100       1 cloudprovisioner.go:398]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4\" to node \"3791k8s00000000\"." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
... skipping 17 lines ...
I0922 04:31:57.515256       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4\" to node \"3791k8s00000001\"." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:31:57.545475       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0922 04:31:57.545646       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-jqbn4-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8320","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"}
I0922 04:31:57.546137       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=3791k8s00000001,disk.csi.azure.com/volume-name!=pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:31:57.555373       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 8 milliseconds
I0922 04:31:57.557514       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39-3791k8s00000001-attachment)
I0922 04:31:57.559538       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.573617       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39-3791k8s00000001-attachment/status 200 OK in 15 milliseconds
I0922 04:31:57.575527       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:31:57.577680       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=19586076
I0922 04:31:57.577873       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:31:57.599256       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 04:31:57.601273       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023061831 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:57.601543       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:31:57.606587       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
... skipping 14 lines ...
I0922 04:31:58.453457       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e-3791k8s00000000-attachment/status 200 OK in 21 milliseconds
I0922 04:31:58.454300       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=22063986
I0922 04:31:58.457428       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4-3791k8s00000000-attachment/status 200 OK in 25 milliseconds
I0922 04:31:58.459107       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4-3791k8s00000000-attachment/status 200 OK in 27 milliseconds
I0922 04:31:58.462236       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=30540265
I0922 04:31:58.462844       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=31547610
I0922 04:31:58.465770       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:58.466055       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=1167740481
I0922 04:31:58.466235       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=1168088297
I0922 04:31:58.466422       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=1223066053
I0922 04:31:58.466540       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.22322986 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" node="3791k8s00000000" result_code="succeeded"
I0922 04:31:58.466561       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0922 04:31:58.471570       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:58.471730       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=1187751376
I0922 04:31:58.471849       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=1188039588
I0922 04:31:58.471952       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=1237781911
I0922 04:31:58.472084       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.238027722 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" node="3791k8s00000000" result_code="succeeded"
I0922 04:31:58.472141       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0922 04:31:58.474437       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:58.474907       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=1177658824
I0922 04:31:58.475162       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=1178041041
I0922 04:31:58.475425       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=1219167279
I0922 04:31:58.475693       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.219463392 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" node="3791k8s00000000" result_code="succeeded"
I0922 04:31:58.475890       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 04:31:58.480250       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 5 lines ...
I0922 04:31:58.482148       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0922 04:31:58.515719       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-hpczfw5p|3791k8s00000001" delay="1s"
I0922 04:31:58.515986       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:31:58.533908       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017864698 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:31:58.533946       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:31:58.534048       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:31:58.548480       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:58.548678       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=1179082488
I0922 04:31:58.548853       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=1179303498
I0922 04:31:58.548963       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=1198048235
I0922 04:31:58.549061       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.198182042 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4" node="3791k8s00000001" result_code="succeeded"
I0922 04:31:58.549082       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0922 04:31:58.551239       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7e5c622e-5895-4293-b7bf-415f682233a4-3791k8s00000001-attachment/status 200 OK in 16 milliseconds
... skipping 3 lines ...
I0922 04:31:58.552334       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4:AttachDiskOptions{diskName: "pvc-7e5c622e-5895-4293-b7bf-415f682233a4", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7:AttachDiskOptions{diskName: "pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39:AttachDiskOptions{diskName: "pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39", lun: 2}])
I0922 04:31:58.553375       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=18940346
I0922 04:31:58.557857       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7-3791k8s00000001-attachment/status 200 OK in 23 milliseconds
I0922 04:31:58.560611       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39-3791k8s00000001-attachment/status 200 OK in 25 milliseconds
I0922 04:31:58.561632       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=27291919
I0922 04:31:58.566309       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=31568910
I0922 04:31:58.569811       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:58.569850       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:31:58.570073       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=1311610810
I0922 04:31:58.570155       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=1311769117
I0922 04:31:58.570305       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=1369496797
I0922 04:31:58.570458       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.369726107 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" node="3791k8s00000001" result_code="succeeded"
I0922 04:31:58.570525       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 04:31:58.570852       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=1013009367
I0922 04:31:58.571096       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=1013482488
I0922 04:31:58.571224       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=1025211112
I0922 04:31:58.571380       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.02540412 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" node="3791k8s00000001" result_code="succeeded"
I0922 04:31:58.571416       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0922 04:31:58.715194       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.265727574 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 04:31:58.715228       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4:AttachDiskOptions{diskName: "pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e:AttachDiskOptions{diskName: "pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4:AttachDiskOptions{diskName: "pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 04:31:59.270853       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.718262595 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 04:31:59.270890       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4:AttachDiskOptions{diskName: "pvc-7e5c622e-5895-4293-b7bf-415f682233a4", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7:AttachDiskOptions{diskName: "pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39:AttachDiskOptions{diskName: "pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39", lun: 2}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 04:31:59.611806       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:31:59.615716       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:31:59.617092       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:32:01.043030       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:32:01.043059       1 utils.go:79] GRPC request: {}
I0922 04:32:01.043102       1 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 906 lines ...
I0922 04:37:10.828835       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7-3791k8s00000001-attachment 200 OK in 33 milliseconds
I0922 04:37:10.831518       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7) scheduled in 5m0s." "disk.csi.azure.com/request-id"="390c7d9a-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:37:10.832632       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7-3791k8s00000001-attachment)
I0922 04:37:10.838650       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39-3791k8s00000001-attachment 200 OK in 24 milliseconds
I0922 04:37:10.841306       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39) scheduled in 5m0s." "disk.csi.azure.com/request-id"="390dfb31-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:37:10.841856       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39-3791k8s00000001-attachment)
I0922 04:37:10.842554       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:10.851860       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7-3791k8s00000001-attachment/status 200 OK in 20 milliseconds
I0922 04:37:10.851863       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:10.853134       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=21573535
I0922 04:37:10.853266       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:37:10.853569       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7 from node 3791k8s00000001" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:37:10.853624       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:37:10.858635       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7e5c622e-5895-4293-b7bf-415f682233a4-3791k8s00000001-attachment 200 OK in 30 milliseconds
I0922 04:37:10.859188       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7e5c622e-5895-4293-b7bf-415f682233a4-3791k8s00000001-attachment)
I0922 04:37:10.859765       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-7e5c622e-5895-4293-b7bf-415f682233a4) scheduled in 5m0s." "disk.csi.azure.com/request-id"="3910cde0-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:37:10.860124       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:10.864814       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39-3791k8s00000001-attachment/status 200 OK in 23 milliseconds
I0922 04:37:10.865483       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:10.866947       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=27598196
I0922 04:37:10.867226       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:37:10.867650       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39 from node 3791k8s00000001" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:37:10.895576       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:10.899446       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7e5c622e-5895-4293-b7bf-415f682233a4-3791k8s00000001-attachment/status 200 OK in 41 milliseconds
I0922 04:37:10.900422       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=42564644
I0922 04:37:10.900474       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:37:10.900860       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4 from node 3791k8s00000001" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:37:10.916448       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.06275852 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:37:10.916478       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
... skipping 59 lines ...
I0922 04:37:27.914522       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=25822546
I0922 04:37:27.914730       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=17014148272
I0922 04:37:27.914928       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=17057163036
I0922 04:37:27.915684       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=26268466
I0922 04:37:27.915914       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=17048409958
I0922 04:37:27.916118       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=17076789588
I0922 04:37:27.916974       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:37:27.917217       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=17075123017
I0922 04:37:27.917430       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e50e926-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=17075580237
I0922 04:37:27.917637       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="3906a529-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=17124447955
I0922 04:37:27.917897       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.125573303 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" node="3791k8s00000001" result_code="succeeded"
I0922 04:37:27.918069       1 utils.go:85] GRPC response: {}
I0922 04:37:27.919589       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:37:27.919873       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=17060410582
I0922 04:37:27.920110       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e3324e7-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=17060942106
I0922 04:37:27.921066       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="39076d61-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=17122764985
I0922 04:37:27.921318       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.12308 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4" node="3791k8s00000001" result_code="succeeded"
I0922 04:37:27.921536       1 utils.go:85] GRPC response: {}
I0922 04:37:27.923862       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:37:27.924216       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=17091495834
I0922 04:37:27.924458       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="7e1c3c5a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=17091838749
I0922 04:37:27.924727       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="3903cd01-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=17150190178
I0922 04:37:27.925017       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.150734701 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" node="3791k8s00000001" result_code="succeeded"
I0922 04:37:27.925226       1 utils.go:85] GRPC response: {}
I0922 04:37:27.954217       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 29 lines ...
I0922 04:37:32.217271       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39)
I0922 04:37:32.218073       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="45c35245-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=55727675
I0922 04:37:32.218104       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-7e5c622e-5895-4293-b7bf-415f682233a4)
I0922 04:37:32.221417       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0922 04:37:32.221596       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"}
I0922 04:37:32.221733       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7)
I0922 04:37:32.233441       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:32.239658       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39 200 OK in 22 milliseconds
I0922 04:37:32.247946       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7/status 200 OK in 25 milliseconds
I0922 04:37:32.248435       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7e5c622e-5895-4293-b7bf-415f682233a4 200 OK in 30 milliseconds
I0922 04:37:32.254291       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:32.263152       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39/status 200 OK in 26 milliseconds
I0922 04:37:32.266226       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:32.269565       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="45cc6ca0-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=47510409
I0922 04:37:32.269789       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7)
I0922 04:37:32.272315       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=36456519
I0922 04:37:32.272367       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:37:32.272798       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39)" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:37:32.272894       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39)" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:37:32.273237       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39"
I0922 04:37:32.273414       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=549224
I0922 04:37:32.285013       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:32.286204       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7e5c622e-5895-4293-b7bf-415f682233a4/status 200 OK in 25 milliseconds
I0922 04:37:32.290390       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=29862026
I0922 04:37:32.290972       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:37:32.291303       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-7e5c622e-5895-4293-b7bf-415f682233a4)" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:37:32.292170       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-7e5c622e-5895-4293-b7bf-415f682233a4)" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:37:32.292486       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-7e5c622e-5895-4293-b7bf-415f682233a4." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4"
I0922 04:37:32.292824       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=1520367
I0922 04:37:32.298798       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.025261321 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:37:32.299564       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7 200 OK in 29 milliseconds
I0922 04:37:32.301640       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:32.310437       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 16 milliseconds
I0922 04:37:32.322420       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 11 milliseconds
I0922 04:37:32.322508       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7/status 200 OK in 18 milliseconds
I0922 04:37:32.325319       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:32.326216       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:37:32.327886       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=24078369
I0922 04:37:32.327930       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:37:32.328011       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7)" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:37:32.328256       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7)" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:37:32.328424       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7"
I0922 04:37:32.328622       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=613727
I0922 04:37:32.386983       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.093869768 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:37:32.392736       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.063869536 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:37:32.814225       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:37:32.814369       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:37:32.814543       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:32.814646       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:32.814677       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:32.814772       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:37:33.401926       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:37:33.404095       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:37:33.404097       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:37:34.331108       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 04:37:34.340425       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
... skipping 5 lines ...
I0922 04:37:37.458540       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39
I0922 04:37:37.458748       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=5185071897
I0922 04:37:37.474504       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39 200 OK in 15 milliseconds
I0922 04:37:37.475477       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=16658837
I0922 04:37:37.475539       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=5202933288
I0922 04:37:37.475588       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="7bae49e5-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=5239748322
I0922 04:37:37.476838       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:37:37.476924       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="45c671ff-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=5236548379
I0922 04:37:37.477124       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "disk.csi.azure.com/request-id"="45c671ff-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" "latency"=5294210740
I0922 04:37:37.477154       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39) returned with <nil>
I0922 04:37:37.477258       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.29464706 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-95e430ef-5103-4ba6-b2b7-1a79d5089a39" result_code="succeeded"
I0922 04:37:37.477288       1 utils.go:85] GRPC response: {}
I0922 04:37:37.606337       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.219293789 request="disks_delete" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:37:37.606369       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4
I0922 04:37:37.606441       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=5313328765
I0922 04:37:37.616681       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7e5c622e-5895-4293-b7bf-415f682233a4 200 OK in 9 milliseconds
I0922 04:37:37.618845       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=12337646
I0922 04:37:37.619159       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=5327902809
I0922 04:37:37.619424       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="7ba6fd78-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=5358918686
I0922 04:37:37.622640       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:37:37.622942       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="45c35245-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=5352118384
I0922 04:37:37.623146       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "disk.csi.azure.com/request-id"="45c35245-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-7e5c622e-5895-4293-b7bf-415f682233a4" "latency"=5460726906
I0922 04:37:37.623222       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4) returned with <nil>
I0922 04:37:37.623406       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.461335133 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-7e5c622e-5895-4293-b7bf-415f682233a4" result_code="succeeded"
I0922 04:37:37.623602       1 utils.go:85] GRPC response: {}
I0922 04:37:37.683525       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.290647042 request="disks_delete" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:37:37.683561       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7
I0922 04:37:37.683745       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="7bb0029b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=5354737588
I0922 04:37:37.698666       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:37:37.699107       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="45cc6ca0-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=5394575254
I0922 04:37:37.699228       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "disk.csi.azure.com/request-id"="45cc6ca0-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" "latency"=5477174222
I0922 04:37:37.699303       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7) returned with <nil>
I0922 04:37:37.699412       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.477606442 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7" result_code="succeeded"
I0922 04:37:37.699434       1 utils.go:85] GRPC response: {}
I0922 04:37:37.701061       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-903fac92-5bd0-4847-9b69-3a8d4ce5fdc7 200 OK in 16 milliseconds
... skipping 59 lines ...
I0922 04:37:59.408420       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4-3791k8s00000000-attachment)
I0922 04:37:59.412028       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4-3791k8s00000000-attachment/status 200 OK in 18 milliseconds
I0922 04:37:59.414473       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0922 04:37:59.414504       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"}
I0922 04:37:59.414719       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e-3791k8s00000000-attachment) detachment" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:37:59.420703       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=27654928
I0922 04:37:59.424295       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:59.424420       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4-3791k8s00000000-attachment/status 200 OK in 21 milliseconds
I0922 04:37:59.427494       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=24749799
I0922 04:37:59.427549       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:37:59.427971       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4 from node 3791k8s00000000" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:37:59.428007       1 azure_vmss_cache.go:332] Node 3791k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:37:59.431913       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e-3791k8s00000000-attachment/status 200 OK in 16 milliseconds
... skipping 4 lines ...
I0922 04:37:59.450230       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e) scheduled in 5m0s." "disk.csi.azure.com/request-id"="56071f3d-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:37:59.451420       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e-3791k8s00000000-attachment 200 OK in 18 milliseconds
I0922 04:37:59.457831       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e-3791k8s00000000-attachment)
I0922 04:37:59.458294       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.030242944 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:37:59.458351       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:37:59.464358       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4-3791k8s00000000-attachment/status 200 OK in 24 milliseconds
I0922 04:37:59.464951       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:59.467685       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=26088559
I0922 04:37:59.468037       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:37:59.468230       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4 from node 3791k8s00000000" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:37:59.468258       1 azure_vmss_cache.go:332] Node 3791k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:37:59.472119       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e-3791k8s00000000-attachment/status 200 OK in 21 milliseconds
I0922 04:37:59.472607       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=22099381
I0922 04:37:59.472680       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:37:59.472911       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e from node 3791k8s00000000" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:37:59.474161       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:37:59.505766       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.037456164 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:37:59.505796       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:37:59.506031       1 azure_vmss_cache.go:332] Node 3791k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:37:59.526029       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019937785 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:37:59.526056       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:38:00.458969       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-hpczfw5p|3791k8s00000000" delay="1s"
... skipping 14 lines ...
I0922 04:38:02.546582       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 04:38:02.555731       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0922 04:38:02.555993       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:38:02.815359       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:38:02.815360       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:38:02.815375       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:38:02.815611       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:02.815678       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:02.815699       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:03.402380       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:38:03.404564       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:38:03.404575       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:38:04.564780       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 04:38:04.571403       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 04:38:04.573099       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 38 lines ...
I0922 04:38:17.134132       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=22092380
I0922 04:38:17.134355       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=17706474685
I0922 04:38:17.134556       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=17731834312
I0922 04:38:17.135470       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=23209530
I0922 04:38:17.135676       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=17667565858
I0922 04:38:17.135865       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=17695844713
I0922 04:38:17.137998       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:17.138267       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=17680136515
I0922 04:38:17.138508       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e22ba9a-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=17680674440
I0922 04:38:17.138706       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="5601b324-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=17724012464
I0922 04:38:17.139697       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.725062811 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" node="3791k8s00000000" result_code="succeeded"
I0922 04:38:17.139875       1 utils.go:85] GRPC response: {}
I0922 04:38:17.145056       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:17.145351       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=17699666383
I0922 04:38:17.145563       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e24b559-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=17700140603
I0922 04:38:17.145791       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="55fe5fd3-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=17752878546
I0922 04:38:17.145979       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.753169959 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" node="3791k8s00000000" result_code="succeeded"
I0922 04:38:17.146165       1 utils.go:85] GRPC response: {}
I0922 04:38:17.149700       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:17.149922       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=17741412436
I0922 04:38:17.150145       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="7e215483-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=17741735251
I0922 04:38:17.150334       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000000" "disk.csi.azure.com/request-id"="55f9f908-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=17786281329
I0922 04:38:17.150524       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.78675875 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" node="3791k8s00000000" result_code="succeeded"
I0922 04:38:17.150703       1 utils.go:85] GRPC response: {}
I0922 04:38:17.166569       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 31 lines ...
I0922 04:38:24.912543       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e)
I0922 04:38:24.924658       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4 200 OK in 12 milliseconds
I0922 04:38:24.930593       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0922 04:38:24.930955       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"}
I0922 04:38:24.931073       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4)
I0922 04:38:24.933216       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e/status 200 OK in 19 milliseconds
I0922 04:38:24.934318       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:24.934954       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="65346a11-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=22069481
I0922 04:38:24.934980       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e)
I0922 04:38:24.936393       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:24.952393       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4/status 200 OK in 19 milliseconds
I0922 04:38:24.954661       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:24.954994       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=22596103
I0922 04:38:24.956590       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:38:24.956907       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4)" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:38:24.957308       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4)" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:38:24.958170       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4"
I0922 04:38:24.958292       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=1408962
I0922 04:38:24.964908       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4/status 200 OK in 31 milliseconds
I0922 04:38:24.965346       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="65373e12-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=33879405
I0922 04:38:24.965375       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4)
I0922 04:38:24.972420       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:24.974308       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e 200 OK in 39 milliseconds
I0922 04:38:24.984334       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.025954053 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:24.985001       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e/status 200 OK in 11 milliseconds
I0922 04:38:24.991837       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4 200 OK in 26 milliseconds
I0922 04:38:24.992379       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:24.993694       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:24.994098       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=22072980
I0922 04:38:24.994142       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:38:24.994212       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e)" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:38:24.994250       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e)" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:38:24.994293       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e"
I0922 04:38:24.994343       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=128406
I0922 04:38:25.004547       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4/status 200 OK in 9 milliseconds
I0922 04:38:25.005264       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=10622272
I0922 04:38:25.005319       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:38:25.005410       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4)" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:38:25.005442       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4)" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:38:25.005519       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4"
I0922 04:38:25.005630       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=181609
I0922 04:38:25.008482       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:25.029504       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023735655 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:25.032507       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.038105592 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:26.748425       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:38:26.757097       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0922 04:38:26.757434       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:38:28.770385       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 11 milliseconds
... skipping 2 lines ...
I0922 04:38:30.180256       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.15060957 request="disks_delete" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:30.180534       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4
I0922 04:38:30.180679       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=5174830645
I0922 04:38:30.186749       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.202366969 request="disks_delete" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:30.187104       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4
I0922 04:38:30.187248       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="7bc5fdfe-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=5228886547
I0922 04:38:30.203909       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:30.204024       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="65373e12-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=5210240818
I0922 04:38:30.204075       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="65373e12-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=5272667491
I0922 04:38:30.204092       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4) returned with <nil>
I0922 04:38:30.204131       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.273032007 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" result_code="succeeded"
I0922 04:38:30.204150       1 utils.go:85] GRPC response: {}
I0922 04:38:30.213035       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.180477696 request="disks_delete" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:30.213062       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e
I0922 04:38:30.213107       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="7bae70d0-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=5218701094
I0922 04:38:30.213954       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4 200 OK in 32 milliseconds
I0922 04:38:30.215911       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=35174262
I0922 04:38:30.215960       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=5210588434
I0922 04:38:30.215987       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "disk.csi.azure.com/request-id"="7bba054b-3a2f-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-e43d1df7-dd83-43b4-a3eb-8b1d854725c4" "latency"=5221382012
I0922 04:38:30.216841       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:30.216895       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="6531487c-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=5291632433
I0922 04:38:30.216946       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "disk.csi.azure.com/request-id"="6531487c-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" "latency"=5324586797
I0922 04:38:30.216958       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4) returned with <nil>
I0922 04:38:30.216987       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.32488061 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4" result_code="succeeded"
I0922 04:38:30.217004       1 utils.go:85] GRPC response: {}
I0922 04:38:30.231065       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-86fbd87b-fbb2-45c2-b1dc-c1d23aaafac4 200 OK in 42 milliseconds
I0922 04:38:30.245338       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:30.245639       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="65346a11-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=5270420990
I0922 04:38:30.245840       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "disk.csi.azure.com/request-id"="65346a11-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" "latency"=5332906366
I0922 04:38:30.245864       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e) returned with <nil>
I0922 04:38:30.246054       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.333338585 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e" result_code="succeeded"
I0922 04:38:30.246138       1 utils.go:85] GRPC response: {}
I0922 04:38:30.248368       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ae5399b6-4504-4eed-acfc-c33e6f29110e 200 OK in 34 milliseconds
... skipping 54 lines ...
I0922 04:38:39.027329       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50)
I0922 04:38:39.027672       1 shared_state.go:457]  "msg"="Pod azuredisk-volume-tester-zz8r2-0: Skipping Volume {kube-api-access-tf2vn {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0922 04:38:39.028253       1 shared_state.go:483]  "msg"="Storing pod azuredisk-volume-tester-zz8r2-0 and claim [] to podToClaimsMap map." 
I0922 04:38:39.034269       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "disk.csi.azure.com/request-id"="6d903118-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=58606203
I0922 04:38:39.034577       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "disk.csi.azure.com/request-id"="6d903118-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0"
I0922 04:38:39.034881       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0) is ZRS(Premium_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "disk.csi.azure.com/request-id"="6d903118-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0"
I0922 04:38:39.041203       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:39.045647       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:39.062703       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 53 milliseconds
I0922 04:38:39.063708       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:38:39.064032       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02882758 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:38:39.064444       1 cloudprovisioner.go:246] begin to create disk(pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0) account type(Premium_ZRS) rg(kubetest-hpczfw5p) location() size(256) selectedAvailabilityZone() maxShares(2)
I0922 04:38:39.072475       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50/status 200 OK in 28 milliseconds
I0922 04:38:39.073364       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "disk.csi.azure.com/request-id"="6d945c6d-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=29423007
I0922 04:38:39.073603       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "disk.csi.azure.com/request-id"="6d945c6d-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50"
I0922 04:38:39.074021       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50) is ZRS(Premium_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "disk.csi.azure.com/request-id"="6d945c6d-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50"
I0922 04:38:39.084624       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 74 milliseconds
I0922 04:38:39.085694       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56"
I0922 04:38:39.085916       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-19963529-e901-485c-9886-a19898f82d56)
I0922 04:38:39.085434       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17"
I0922 04:38:39.086349       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-4677ae46-f78d-48a3-9eed-d62d49691b17)
I0922 04:38:39.093644       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:38:39.093906       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019174152 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:38:39.094050       1 cloudprovisioner.go:246] begin to create disk(pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50) account type(Premium_ZRS) rg(kubetest-hpczfw5p) location() size(256) selectedAvailabilityZone() maxShares(2)
I0922 04:38:39.097454       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:39.100953       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:39.101143       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:39.121797       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-19963529-e901-485c-9886-a19898f82d56/status 200 OK in 23 milliseconds
I0922 04:38:39.122558       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028304758 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:39.122646       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50 StorageAccountType:Premium_ZRS Size:256
I0922 04:38:39.124772       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:39.127557       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=30854270
I0922 04:38:39.127665       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56"
I0922 04:38:39.128056       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-19963529-e901-485c-9886-a19898f82d56) is ZRS(Premium_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56"
I0922 04:38:39.136223       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17/status 200 OK in 43 milliseconds
I0922 04:38:39.138610       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:39.140778       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=47983131
I0922 04:38:39.140969       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17"
I0922 04:38:39.141259       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-4677ae46-f78d-48a3-9eed-d62d49691b17) is ZRS(Premium_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17"
I0922 04:38:39.142699       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-19963529-e901-485c-9886-a19898f82d56, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-19963529-e901-485c-9886-a19898f82d56' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:38:39.142944       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.014480644 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:38:39.143123       1 cloudprovisioner.go:246] begin to create disk(pvc-19963529-e901-485c-9886-a19898f82d56) account type(Premium_ZRS) rg(kubetest-hpczfw5p) location() size(256) selectedAvailabilityZone() maxShares(2)
I0922 04:38:39.148380       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.082536567 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:39.148421       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0 StorageAccountType:Premium_ZRS Size:256
I0922 04:38:39.163196       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 04:38:39.163417       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021933774 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 04:38:39.163530       1 cloudprovisioner.go:246] begin to create disk(pvc-4677ae46-f78d-48a3-9eed-d62d49691b17) account type(Premium_ZRS) rg(kubetest-hpczfw5p) location() size(256) selectedAvailabilityZone() maxShares(2)
I0922 04:38:39.183517       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.040326191 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:39.183564       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-19963529-e901-485c-9886-a19898f82d56 StorageAccountType:Premium_ZRS Size:256
I0922 04:38:39.192151       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028520767 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:39.192201       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-4677ae46-f78d-48a3-9eed-d62d49691b17 StorageAccountType:Premium_ZRS Size:256
... skipping 16 lines ...
I0922 04:38:41.475934       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "disk.csi.azure.com/request-id"="6d903118-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=2441209041
I0922 04:38:41.475953       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "disk.csi.azure.com/request-id"="6d903118-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=2500331468
I0922 04:38:41.476238       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "disk.csi.azure.com/request-id"="6d945c6d-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=42113471
I0922 04:38:41.476270       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "disk.csi.azure.com/request-id"="6d945c6d-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=2402419318
I0922 04:38:41.476289       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "disk.csi.azure.com/request-id"="6d945c6d-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=2432395350
I0922 04:38:41.476632       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.284395275 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:41.484027       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:41.484255       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "disk.csi.azure.com/request-id"="6d903118-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=2506920160
I0922 04:38:41.484432       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "disk.csi.azure.com/request-id"="6d903118-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=2548272597
I0922 04:38:41.484607       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.548448404 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" result_code="succeeded"
I0922 04:38:41.484693       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":274877906944,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-qkggm-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-7418","maxShares":"2","requestedsizegib":"256","skuname":"Premium_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0"}}
I0922 04:38:41.488092       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:41.489068       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "disk.csi.azure.com/request-id"="6d945c6d-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=2461260132
I0922 04:38:41.489169       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "disk.csi.azure.com/request-id"="6d945c6d-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=2526313121
I0922 04:38:41.489273       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.5264570280000003 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" result_code="succeeded"
I0922 04:38:41.489296       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":274877906944,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-zz8r2-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-7418","maxShares":"2","requestedsizegib":"256","skuname":"Premium_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50"}}
I0922 04:38:41.500959       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.024278179 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:41.501302       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-4677ae46-f78d-48a3-9eed-d62d49691b17 StorageAccountType:Premium_ZRS Size:256
I0922 04:38:41.501614       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-4677ae46-f78d-48a3-9eed-d62d49691b17) account type(Premium_ZRS) rg(kubetest-hpczfw5p) location() size(256) tags(map[kubernetes.io-created-for-pv-name:pvc-4677ae46-f78d-48a3-9eed-d62d49691b17 kubernetes.io-created-for-pvc-name:test-volume-2-azuredisk-volume-tester-zz8r2-0 kubernetes.io-created-for-pvc-namespace:azuredisk-7418]) successfully" "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17"
I0922 04:38:41.501786       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=2360524657
I0922 04:38:41.538486       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17/status 200 OK in 36 milliseconds
I0922 04:38:41.545604       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.361919119 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:41.550065       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:41.550313       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=2463778843
I0922 04:38:41.550896       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=2542728451
I0922 04:38:41.551197       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.543047465 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" result_code="succeeded"
I0922 04:38:41.551369       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":274877906944,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-4677ae46-f78d-48a3-9eed-d62d49691b17","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-zz8r2-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-7418","maxShares":"2","requestedsizegib":"256","skuname":"Premium_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17"}}
I0922 04:38:41.554121       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=52209219
I0922 04:38:41.554350       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi.azure.com/request-id"="6d9b2ff4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=2413225998
... skipping 3 lines ...
I0922 04:38:41.569329       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-19963529-e901-485c-9886-a19898f82d56) account type(Premium_ZRS) rg(kubetest-hpczfw5p) location() size(256) tags(map[kubernetes.io-created-for-pv-name:pvc-19963529-e901-485c-9886-a19898f82d56 kubernetes.io-created-for-pvc-name:test-volume-2-azuredisk-volume-tester-qkggm-0 kubernetes.io-created-for-pvc-namespace:azuredisk-7418]) successfully" "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56"
I0922 04:38:41.569380       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=2441474253
I0922 04:38:41.593981       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-19963529-e901-485c-9886-a19898f82d56/status 200 OK in 24 milliseconds
I0922 04:38:41.594696       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=25246022
I0922 04:38:41.594811       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=2466962485
I0922 04:38:41.595024       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=2498279476
I0922 04:38:41.595963       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:41.596676       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=2510478918
I0922 04:38:41.597017       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-19963529-e901-485c-9886-a19898f82d56" "disk.csi.azure.com/request-id"="6d9b6645-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=2587204226
I0922 04:38:41.597080       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.587557542 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-19963529-e901-485c-9886-a19898f82d56" result_code="succeeded"
I0922 04:38:41.597102       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":274877906944,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-19963529-e901-485c-9886-a19898f82d56","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-qkggm-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-7418","maxShares":"2","requestedsizegib":"256","skuname":"Premium_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-19963529-e901-485c-9886-a19898f82d56"}}
I0922 04:38:42.490863       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0922 04:38:42.491165       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000003","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-4677ae46-f78d-48a3-9eed-d62d49691b17","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-zz8r2-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-7418","maxShares":"2","requestedsizegib":"256","skuname":"Premium_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17"}
... skipping 15 lines ...
I0922 04:38:42.551450       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 27 milliseconds
I0922 04:38:42.551651       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17-3791k8s00000003-attachment/status 200 OK in 21 milliseconds
I0922 04:38:42.554259       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6faeb5ef-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=24728498
I0922 04:38:42.554317       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6faeb5ef-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17"
I0922 04:38:42.559281       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0-3791k8s00000002-attachment)
I0922 04:38:42.559635       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4677ae46-f78d-48a3-9eed-d62d49691b17-3791k8s00000003-attachment)
I0922 04:38:42.563853       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:42.563897       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:42.564219       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 39 milliseconds
I0922 04:38:42.565852       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 47 milliseconds
I0922 04:38:42.567585       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50-3791k8s00000003-attachment)
I0922 04:38:42.568417       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-19963529-e901-485c-9886-a19898f82d56-3791k8s00000002-attachment)
I0922 04:38:42.571918       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:42.585251       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:42.585419       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0-3791k8s00000002-attachment/status 200 OK in 24 milliseconds
I0922 04:38:42.586336       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb3993a-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=25956053
I0922 04:38:42.586417       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb3993a-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0"
I0922 04:38:42.587823       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.026460776 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:42.589962       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:38:42.591774       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-19963529-e901-485c-9886-a19898f82d56-3791k8s00000002-attachment/status 200 OK in 21 milliseconds
I0922 04:38:42.591883       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:42.598296       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb29fc4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=28396561
I0922 04:38:42.598376       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb29fc4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56"
I0922 04:38:42.604177       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50-3791k8s00000003-attachment/status 200 OK in 33 milliseconds
I0922 04:38:42.604558       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6fb358b7-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=33629994
I0922 04:38:42.604626       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6fb358b7-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50"
I0922 04:38:42.606080       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:38:42.611622       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02160796 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:42.611652       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:38:42.611722       1 cloudprovisioner.go:398]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17\" to node \"3791k8s00000003\"." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6faeb5ef-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17"
I0922 04:38:42.611771       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17\" to node \"3791k8s00000003\"." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6faeb5ef-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17"
I0922 04:38:42.623271       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.024668496 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:42.623333       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
... skipping 20 lines ...
I0922 04:38:43.613006       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:38:43.654715       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.04164995 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:43.655439       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:38:43.655505       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:38:43.671342       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17-3791k8s00000003-attachment/status 200 OK in 15 milliseconds
I0922 04:38:43.673183       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6faeb5ef-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=17587081
I0922 04:38:43.673762       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:43.673867       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-hpczfw5p|3791k8s00000002" delay="1s"
I0922 04:38:43.673947       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6faeb5ef-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=1114103089
I0922 04:38:43.674043       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6faeb5ef-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=1114389703
I0922 04:38:43.674140       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6faeb5ef-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=1182513828
I0922 04:38:43.674210       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.182655134 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" node="3791k8s00000003" result_code="succeeded"
I0922 04:38:43.674227       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 04:38:43.676491       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020917429 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:43.676515       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:38:43.676530       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17:AttachDiskOptions{diskName: "pvc-4677ae46-f78d-48a3-9eed-d62d49691b17", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50:AttachDiskOptions{diskName: "pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50", lun: 1}]
I0922 04:38:43.676612       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17:AttachDiskOptions{diskName: "pvc-4677ae46-f78d-48a3-9eed-d62d49691b17", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50:AttachDiskOptions{diskName: "pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50", lun: 1}])
I0922 04:38:43.677316       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:38:43.678350       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50-3791k8s00000003-attachment/status 200 OK in 22 milliseconds
I0922 04:38:43.679432       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:43.680129       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6fb358b7-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=24356782
I0922 04:38:43.680243       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6fb358b7-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=1112524920
I0922 04:38:43.680331       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6fb358b7-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=1112729529
I0922 04:38:43.680369       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="6fb358b7-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=1158420759
I0922 04:38:43.681268       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.158521162 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" node="3791k8s00000003" result_code="succeeded"
I0922 04:38:43.681292       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
... skipping 10 lines ...
I0922 04:38:43.702034       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 04:38:43.720420       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-19963529-e901-485c-9886-a19898f82d56-3791k8s00000002-attachment/status 200 OK in 18 milliseconds
I0922 04:38:43.721873       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019293257 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 04:38:43.722175       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 04:38:43.723026       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-19963529-e901-485c-9886-a19898f82d56:AttachDiskOptions{diskName: "pvc-19963529-e901-485c-9886-a19898f82d56", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0:AttachDiskOptions{diskName: "pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0", lun: 1}]
I0922 04:38:43.723552       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-19963529-e901-485c-9886-a19898f82d56:AttachDiskOptions{diskName: "pvc-19963529-e901-485c-9886-a19898f82d56", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0:AttachDiskOptions{diskName: "pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0", lun: 1}])
I0922 04:38:43.722935       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:43.724707       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb29fc4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=26052257
I0922 04:38:43.725711       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb29fc4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=1156757284
I0922 04:38:43.725992       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb29fc4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=1157737728
I0922 04:38:43.726270       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb29fc4-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=1208996105
I0922 04:38:43.726564       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.209357621 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-19963529-e901-485c-9886-a19898f82d56" node="3791k8s00000002" result_code="succeeded"
I0922 04:38:43.726758       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 04:38:43.733905       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0-3791k8s00000002-attachment/status 200 OK in 31 milliseconds
I0922 04:38:43.735768       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:38:43.736951       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb3993a-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=1177528807
I0922 04:38:43.737029       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb3993a-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=1177784418
I0922 04:38:43.737058       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb3993a-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=35074958
I0922 04:38:43.737065       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6fb3993a-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=1213465903
I0922 04:38:43.737431       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.213824919 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" node="3791k8s00000002" result_code="succeeded"
I0922 04:38:43.737454       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
... skipping 3 lines ...
I0922 04:38:43.741245       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name!=3791k8s00000002,disk.csi.azure.com/volume-name=pvc-19963529-e901-485c-9886-a19898f82d56." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="706d5b5e-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56"
I0922 04:38:43.741347       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="706d5b5e-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=43102
I0922 04:38:43.741395       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="706d5b5e-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=407618
I0922 04:38:43.741436       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.000492622 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-19963529-e901-485c-9886-a19898f82d56" node="3791k8s00000002" result_code="succeeded"
I0922 04:38:43.741455       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 04:38:44.500289       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.776446391 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 04:38:44.500559       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-19963529-e901-485c-9886-a19898f82d56:AttachDiskOptions{diskName: "pvc-19963529-e901-485c-9886-a19898f82d56", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0:AttachDiskOptions{diskName: "pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 04:38:44.900169       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:38:44.903633       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:38:44.904246       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:38:45.208525       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.531863445 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 04:38:45.208562       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-4677ae46-f78d-48a3-9eed-d62d49691b17:AttachDiskOptions{diskName: "pvc-4677ae46-f78d-48a3-9eed-d62d49691b17", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50:AttachDiskOptions{diskName: "pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 04:38:46.907933       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 04:38:46.912151       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 04:38:46.912310       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 04:38:48.917823       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 04:38:48.924136       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 04:38:48.924301       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 173 lines ...
I0922 04:39:49.132133       1 shared_state.go:588]  "msg"="Getting ranked list of nodes for creating AzVolumeAttachments" "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
I0922 04:39:49.132330       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-4677ae46-f78d-48a3-9eed-d62d49691b17) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
I0922 04:39:49.132408       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-4677ae46-f78d-48a3-9eed-d62d49691b17)" "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
I0922 04:39:49.132502       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/requested-role=Primary,disk.csi.azure.com/volume-name=pvc-4677ae46-f78d-48a3-9eed-d62d49691b17." "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
I0922 04:39:49.132576       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
I0922 04:39:49.132627       1 shared_state.go:783]  "msg"="The list of zones to select nodes from is: westus2-2,westus2-3,westus2-1" "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
I0922 04:39:49.132674       1 shared_state.go:795]  "msg"="failed to find zone annotations for primary node" "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
I0922 04:39:49.132974       1 common.go:394]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0" "filter-plugin"="inter-pod affinity filter" "latency"=13501
I0922 04:39:49.133012       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod affinity filter): [3791k8s00000000 k8s-master-37910252-vmss000000 3791k8s00000002]" "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
I0922 04:39:49.133056       1 common.go:511]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAntiAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0" "filter-plugin"="inter-pod anti-affinity filter" "latency"=9500
I0922 04:39:49.133088       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod anti-affinity filter): [k8s-master-37910252-vmss000000 3791k8s00000002 3791k8s00000000]" "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
I0922 04:39:49.133137       1 common.go:582]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*podTolerationFilter).filter" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0" "filter-plugin"="pod toleration filter" "latency"=10200
I0922 04:39:49.133169       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (pod toleration filter): [k8s-master-37910252-vmss000000 3791k8s00000002 3791k8s00000000]" "disk.csi.azure.com/request-id"="976721de-3a30-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "disk.csi/azure.com/pod-name"="azuredisk-7418/azuredisk-volume-tester-zz8r2-0"
... skipping 9047 lines ...
I0922 05:16:59.227052       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0922 05:16:59.227090       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-fc274787-ab7a-4751-9d73-f934d576cb50","parameters":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-fc274787-ab7a-4751-9d73-f934d576cb50","csi.storage.k8s.io/pvc/name":"pvc-7hsct","csi.storage.k8s.io/pvc/namespace":"azuredisk-741","maxshares":"2","skuname":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0922 05:16:59.228307       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:16:59.246802       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 17 milliseconds
I0922 05:16:59.256610       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:16:59.256795       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-fc274787-ab7a-4751-9d73-f934d576cb50)
I0922 05:16:59.274978       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:16:59.276129       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-fc274787-ab7a-4751-9d73-f934d576cb50/status 200 OK in 19 milliseconds
I0922 05:16:59.277128       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=21158434
I0922 05:16:59.277560       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:16:59.438436       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 05:16:59.438780       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.160643087 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 05:16:59.438843       1 cloudprovisioner.go:246] begin to create disk(pvc-fc274787-ab7a-4751-9d73-f934d576cb50) account type(Premium_LRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(2)
I0922 05:16:59.512658       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.073382138 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:16:59.513031       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-fc274787-ab7a-4751-9d73-f934d576cb50 StorageAccountType:Premium_LRS Size:10
I0922 05:17:01.044032       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 05:17:01.044059       1 utils.go:79] GRPC request: {}
... skipping 4 lines ...
I0922 05:17:01.703955       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.190668549 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:17:01.724573       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020547607 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:17:01.724612       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-fc274787-ab7a-4751-9d73-f934d576cb50 StorageAccountType:Premium_LRS Size:10
I0922 05:17:01.724678       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-fc274787-ab7a-4751-9d73-f934d576cb50) account type(Premium_LRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-fc274787-ab7a-4751-9d73-f934d576cb50 kubernetes.io-created-for-pvc-name:pvc-7hsct kubernetes.io-created-for-pvc-namespace:azuredisk-741]) successfully" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:01.724968       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=2447160066
I0922 05:17:01.740197       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-fc274787-ab7a-4751-9d73-f934d576cb50/status 200 OK in 14 milliseconds
I0922 05:17:01.742626       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:17:01.743178       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=18139901
I0922 05:17:01.743260       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=2465465273
I0922 05:17:01.743302       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=2487360839
I0922 05:17:01.743531       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=2485612362
I0922 05:17:01.743704       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=2515481580
I0922 05:17:01.743921       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.515875998 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50" result_code="succeeded"
... skipping 4 lines ...
I0922 05:17:02.344398       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name!=3791k8s00000003,disk.csi.azure.com/volume-name=pvc-fc274787-ab7a-4751-9d73-f934d576cb50." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:02.352993       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 8 milliseconds
I0922 05:17:02.353381       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000003-attachment)
I0922 05:17:02.358983       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000003-attachment/status 200 OK in 5 milliseconds
I0922 05:17:02.359410       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=6536788
I0922 05:17:02.359461       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:02.360261       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:17:02.383611       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023965457 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:17:02.383705       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:17:02.401110       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017366266 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:17:02.401143       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:17:02.401282       1 cloudprovisioner.go:398]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50\" to node \"3791k8s00000003\"." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:02.401334       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50\" to node \"3791k8s00000003\"." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:02.881170       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:17:02.881171       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:17:02.881185       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:17:02.881368       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:17:03.067362       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 05:17:03.073298       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:17:03.075280       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:17:03.402259       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-hpczfw5p|3791k8s00000003" delay="1s"
I0922 05:17:03.402350       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:17:03.420001       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017599676 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:17:03.420028       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:17:03.420074       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:17:03.428381       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000003-attachment/status 200 OK in 7 milliseconds
I0922 05:17:03.430375       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=10053644
I0922 05:17:03.433838       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:17:03.434077       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=1080447368
I0922 05:17:03.434268       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=1080755082
I0922 05:17:03.434359       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=1090399207
I0922 05:17:03.434490       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.090633417 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50" node="3791k8s00000003" result_code="succeeded"
I0922 05:17:03.434628       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 05:17:03.436613       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016491927 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:17:03.436644       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:17:03.436682       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50:AttachDiskOptions{diskName: "pvc-fc274787-ab7a-4751-9d73-f934d576cb50", lun: 0}]
I0922 05:17:03.436808       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50:AttachDiskOptions{diskName: "pvc-fc274787-ab7a-4751-9d73-f934d576cb50", lun: 0}])
I0922 05:17:03.469783       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:17:03.469783       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:17:03.469798       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:17:03.571805       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.134934453 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 05:17:03.571850       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50:AttachDiskOptions{diskName: "pvc-fc274787-ab7a-4751-9d73-f934d576cb50", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 05:17:05.083160       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 05:17:05.088503       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 05:17:05.090742       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:17:07.101206       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0922 05:17:07.104792       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 05:17:07.104951       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 76 lines ...
I0922 05:17:34.746120       1 shared_state.go:330]  "msg"="Requested volumes for pod azuredisk-741/azuredisk-volume-tester-rxddk are now the following: Volumes: [pvc-fc274787-ab7a-4751-9d73-f934d576cb50], Len: 1" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
I0922 05:17:34.746165       1 shared_state.go:588]  "msg"="Getting ranked list of nodes for creating AzVolumeAttachments" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
I0922 05:17:34.746506       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-fc274787-ab7a-4751-9d73-f934d576cb50) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
I0922 05:17:34.746573       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-fc274787-ab7a-4751-9d73-f934d576cb50)" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
I0922 05:17:34.746762       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/requested-role=Primary,disk.csi.azure.com/volume-name=pvc-fc274787-ab7a-4751-9d73-f934d576cb50." "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
I0922 05:17:34.746866       1 shared_state.go:783]  "msg"="The list of zones to select nodes from is: westus2-2" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
I0922 05:17:34.746960       1 shared_state.go:795]  "msg"="failed to find zone annotations for primary node" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
I0922 05:17:34.747336       1 common.go:394]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk" "filter-plugin"="inter-pod affinity filter" "latency"=11901
I0922 05:17:34.747482       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod affinity filter): [3791k8s00000003 3791k8s00000001]" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
I0922 05:17:34.747604       1 common.go:511]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAntiAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk" "filter-plugin"="inter-pod anti-affinity filter" "latency"=47002
I0922 05:17:34.747644       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod anti-affinity filter): [3791k8s00000003 3791k8s00000001]" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
I0922 05:17:34.747761       1 common.go:582]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*podTolerationFilter).filter" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk" "filter-plugin"="pod toleration filter" "latency"=44602
I0922 05:17:34.747798       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (pod toleration filter): [3791k8s00000003 3791k8s00000001]" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi/azure.com/pod-name"="azuredisk-741/azuredisk-volume-tester-rxddk"
... skipping 50 lines ...
I0922 05:17:35.884100       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=12407949
I0922 05:17:35.888127       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016635735 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:17:35.888155       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:17:35.888501       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50:AttachDiskOptions{diskName: "pvc-fc274787-ab7a-4751-9d73-f934d576cb50", lun: 0}]
I0922 05:17:35.888709       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50:AttachDiskOptions{diskName: "pvc-fc274787-ab7a-4751-9d73-f934d576cb50", lun: 0}])
I0922 05:17:36.060966       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.172201808 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 05:17:36.061002       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50:AttachDiskOptions{diskName: "pvc-fc274787-ab7a-4751-9d73-f934d576cb50", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 05:17:37.342165       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 14 milliseconds
I0922 05:17:37.347767       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:17:37.347924       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:17:39.361146       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 05:17:39.367344       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:17:39.367927       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 54 lines ...
I0922 05:17:57.007294       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-fc274787-ab7a-4751-9d73-f934d576cb50)" "disk.csi.azure.com/request-id"="eb1546c1-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:57.007476       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/requested-role=Replica,disk.csi.azure.com/volume-name=pvc-fc274787-ab7a-4751-9d73-f934d576cb50." "disk.csi.azure.com/request-id"="eb1546c1-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:57.024505       1 round_trippers.go:553] PATCH https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000003-attachment/status 200 OK in 15 milliseconds
I0922 05:17:57.032816       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-fc274787-ab7a-4751-9d73-f934d576cb50/status 200 OK in 26 milliseconds
I0922 05:17:57.033635       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="eb152bab-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=27085597
I0922 05:17:57.033686       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-fc274787-ab7a-4751-9d73-f934d576cb50)
I0922 05:17:57.042599       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:17:57.048359       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000003-attachment 200 OK in 15 milliseconds
I0922 05:17:57.052616       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-fc274787-ab7a-4751-9d73-f934d576cb50 200 OK in 18 milliseconds
I0922 05:17:57.057549       1 shared_state.go:1028]  "msg"="Set deletion timestamp for AzVolumeAttachment (pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000003-attachment)" "disk.csi.azure.com/request-id"="eb1546c1-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:57.061925       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:17:57.077713       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000003-attachment/status 200 OK in 19 milliseconds
I0922 05:17:57.079843       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=24167767
I0922 05:17:57.080249       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:57.080529       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50 from node 3791k8s00000003" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:57.080734       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:17:57.084166       1 round_trippers.go:553] PATCH https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000001-attachment/status 200 OK in 24 milliseconds
I0922 05:17:57.089348       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-fc274787-ab7a-4751-9d73-f934d576cb50/status 200 OK in 28 milliseconds
I0922 05:17:57.090829       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:17:57.094410       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=33421076
I0922 05:17:57.094718       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:57.095014       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-fc274787-ab7a-4751-9d73-f934d576cb50)" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:57.095323       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-fc274787-ab7a-4751-9d73-f934d576cb50)" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:57.095538       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-fc274787-ab7a-4751-9d73-f934d576cb50." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:17:57.100086       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000001-attachment 200 OK in 12 milliseconds
... skipping 40 lines ...
I0922 05:18:01.044194       1 utils.go:79] GRPC request: {}
I0922 05:18:01.044495       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 05:18:01.544036       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 05:18:01.550600       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:18:01.550953       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:18:02.883166       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:18:02.883285       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:18:02.883167       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:18:02.883183       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:18:03.470546       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:18:03.470547       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:18:03.471668       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:18:03.471784       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:18:03.472004       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:18:03.560423       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0922 05:18:03.567068       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:18:03.567584       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:18:05.576526       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0922 05:18:05.583814       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:18:05.584317       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 3 lines ...
I0922 05:18:09.606367       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 05:18:09.621587       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 14 milliseconds
I0922 05:18:09.623414       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:18:11.631216       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 05:18:11.635975       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 05:18:11.636928       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
E0922 05:18:12.006154       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="eb152bab-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=14945844207
E0922 05:18:12.006674       1 crdprovisioner.go:368]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context canceled]" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="eb152bab-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=15000085003
I0922 05:18:12.006711       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50) returned with context canceled
I0922 05:18:12.006874       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=15.000543123 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50" result_code="failed"
E0922 05:18:12.006906       1 utils.go:83] GRPC error: context canceled
I0922 05:18:13.007088       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0922 05:18:13.007365       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50"}
I0922 05:18:13.007702       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50)
I0922 05:18:13.008235       1 azure_disk_utils.go:1122]  "msg"="Skip update. No update needed." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="f49ed203-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:18:13.008303       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="f49ed203-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=229710
I0922 05:18:13.008376       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-fc274787-ab7a-4751-9d73-f934d576cb50)
... skipping 4 lines ...
I0922 05:18:13.479267       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50 from node 3791k8s00000001 successfully" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:18:13.479396       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=16355084930
I0922 05:18:13.497125       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000001-attachment 200 OK in 17 milliseconds
I0922 05:18:13.497712       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=18205016
I0922 05:18:13.497767       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=16373476553
I0922 05:18:13.497914       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="ddd051aa-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=16396626576
I0922 05:18:13.501112       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:18:13.501355       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4.5" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=16362705080
I0922 05:18:13.646522       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0922 05:18:13.653317       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:18:13.653942       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:18:14.350535       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.119692405 request="vmssvm_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="detach_disk" error_code=""
I0922 05:18:14.350890       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50:pvc-fc274787-ab7a-4751-9d73-f934d576cb50]) returned with <nil>
I0922 05:18:14.351116       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 3791k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50:pvc-fc274787-ab7a-4751-9d73-f934d576cb50]
I0922 05:18:14.351529       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-fc274787-ab7a-4751-9d73-f934d576cb50, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50) succeeded
I0922 05:18:14.351793       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50 from node 3791k8s00000003 successfully" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50"
I0922 05:18:14.352012       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=17271525687
I0922 05:18:14.390920       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000003-attachment 200 OK in 38 milliseconds
I0922 05:18:14.391549       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:18:14.391724       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=39184756
I0922 05:18:14.392020       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=17311518079
I0922 05:18:14.391735       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4.5" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=17252902990
I0922 05:18:14.392493       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="ca805490-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=17336570387
I0922 05:18:14.422483       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.030063248 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:18:15.660776       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
... skipping 6 lines ...
I0922 05:18:19.564361       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50
I0922 05:18:19.564556       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=5172112841
I0922 05:18:19.578441       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-fc274787-ab7a-4751-9d73-f934d576cb50 200 OK in 13 milliseconds
I0922 05:18:19.580809       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=15936406
I0922 05:18:19.581046       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=22486082332
I0922 05:18:19.581264       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="c8a4eae8-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=22520296843
I0922 05:18:19.583676       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:18:19.584429       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="f49ed203-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=6575894965
I0922 05:18:19.584651       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "disk.csi.azure.com/request-id"="f49ed203-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=6576542895
I0922 05:18:19.584781       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50) returned with <nil>
I0922 05:18:19.585003       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=6.577201524 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50" result_code="succeeded"
I0922 05:18:19.585132       1 utils.go:85] GRPC response: {}
I0922 05:18:19.693694       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 10 milliseconds
... skipping 12 lines ...
I0922 05:18:24.017004       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0922 05:18:24.017032       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-f426f608-0ffc-486a-afb4-0aca226d9da0","parameters":{"csi.storage.k8s.io/pv/name":"pvc-f426f608-0ffc-486a-afb4-0aca226d9da0","csi.storage.k8s.io/pvc/name":"pvc1-azuredisk-volume-tester-4w6fj-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1959","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0922 05:18:24.017619       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:18:24.035531       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 17 milliseconds
I0922 05:18:24.038337       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:18:24.038509       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-f426f608-0ffc-486a-afb4-0aca226d9da0)
I0922 05:18:24.046538       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:18:24.047206       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0/status 200 OK in 10 milliseconds
I0922 05:18:24.048426       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=11698628
I0922 05:18:24.048519       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:18:24.081175       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 05:18:24.081305       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.03233876 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 05:18:24.081445       1 cloudprovisioner.go:246] begin to create disk(pvc-f426f608-0ffc-486a-afb4-0aca226d9da0) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0922 05:18:24.116230       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.034727768 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:18:24.116272       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-f426f608-0ffc-486a-afb4-0aca226d9da0 StorageAccountType:StandardSSD_LRS Size:10
I0922 05:18:25.737011       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:18:25.742243       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 05:18:25.744360       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:18:26.433499       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.317179528 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:18:26.456374       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02280173 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:18:26.456410       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-f426f608-0ffc-486a-afb4-0aca226d9da0 StorageAccountType:StandardSSD_LRS Size:10
I0922 05:18:26.456476       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-f426f608-0ffc-486a-afb4-0aca226d9da0) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-f426f608-0ffc-486a-afb4-0aca226d9da0 kubernetes.io-created-for-pvc-name:pvc1-azuredisk-volume-tester-4w6fj-0 kubernetes.io-created-for-pvc-namespace:azuredisk-1959]) successfully" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:18:26.456525       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=2407632612
I0922 05:18:26.472581       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0/status 200 OK in 15 milliseconds
I0922 05:18:26.473634       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:18:26.473828       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=2435077651
I0922 05:18:26.473951       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=2456404914
I0922 05:18:26.474084       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.456616924 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" result_code="succeeded"
I0922 05:18:26.474186       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f426f608-0ffc-486a-afb4-0aca226d9da0","csi.storage.k8s.io/pvc/name":"pvc1-azuredisk-volume-tester-4w6fj-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1959","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"}}
I0922 05:18:26.477707       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=21125053
I0922 05:18:26.477879       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=2428931573
I0922 05:18:26.478002       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=2441279631
I0922 05:18:27.060537       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0922 05:18:27.060568       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000003","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f426f608-0ffc-486a-afb4-0aca226d9da0","csi.storage.k8s.io/pvc/name":"pvc1-azuredisk-volume-tester-4w6fj-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1959","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"}
I0922 05:18:27.061295       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=3791k8s00000003,disk.csi.azure.com/volume-name!=pvc-f426f608-0ffc-486a-afb4-0aca226d9da0." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:18:27.070909       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 9 milliseconds
I0922 05:18:27.074404       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment)
I0922 05:18:27.074765       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:18:27.090535       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment/status 200 OK in 21 milliseconds
I0922 05:18:27.090939       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:18:27.091193       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=21883489
I0922 05:18:27.091395       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:18:27.125963       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.034288949 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:18:27.126035       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:18:27.148722       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022617121 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:18:27.148786       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
... skipping 8 lines ...
I0922 05:18:28.251440       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:18:28.271092       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019603085 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:18:28.271316       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:18:28.271499       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:18:28.280878       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment/status 200 OK in 9 milliseconds
I0922 05:18:28.281640       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=10047454
I0922 05:18:28.283383       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:18:28.283561       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=1208956888
I0922 05:18:28.283673       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=1209245701
I0922 05:18:28.283766       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=1222562002
I0922 05:18:28.283868       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.222791413 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" node="3791k8s00000003" result_code="succeeded"
I0922 05:18:28.283888       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 05:18:28.290241       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018585639 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:18:28.290436       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:18:28.290597       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0:AttachDiskOptions{diskName: "pvc-f426f608-0ffc-486a-afb4-0aca226d9da0", lun: 0}]
I0922 05:18:28.290790       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0:AttachDiskOptions{diskName: "pvc-f426f608-0ffc-486a-afb4-0aca226d9da0", lun: 0}])
I0922 05:18:28.441597       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.150615 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 05:18:28.441790       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0:AttachDiskOptions{diskName: "pvc-f426f608-0ffc-486a-afb4-0aca226d9da0", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 05:18:29.762766       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 05:18:29.776030       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 12 milliseconds
I0922 05:18:29.776934       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:18:31.043859       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 05:18:31.043885       1 utils.go:79] GRPC request: {}
I0922 05:18:31.043949       1 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 244 lines ...
I0922 05:20:17.786118       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment) detachment" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:20:17.796939       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment/status 200 OK in 10 milliseconds
I0922 05:20:17.798475       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=12295160
I0922 05:20:17.805444       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment 200 OK in 6 milliseconds
I0922 05:20:17.805742       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment)
I0922 05:20:17.807179       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-f426f608-0ffc-486a-afb4-0aca226d9da0) scheduled in 5m0s." "disk.csi.azure.com/request-id"="3f01a654-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:20:17.808443       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:20:17.816287       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:20:17.820230       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment/status 200 OK in 13 milliseconds
I0922 05:20:17.820917       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=14182247
I0922 05:20:17.820989       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:20:17.821176       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0 from node 3791k8s00000003" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:20:17.821206       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:20:17.871477       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.05022549 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 32 lines ...
I0922 05:20:29.265294       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0 from node 3791k8s00000003 successfully" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:20:29.265524       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=11444460577
I0922 05:20:29.282221       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment 200 OK in 16 milliseconds
I0922 05:20:29.282560       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=16800891
I0922 05:20:29.282631       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=11461582481
I0922 05:20:29.282670       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=11475967436
I0922 05:20:29.283727       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:20:29.283815       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=11477968922
I0922 05:20:29.283864       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fcff2753-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=11478160631
I0922 05:20:29.283906       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="3efe7035-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=11497813427
I0922 05:20:29.283953       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=11.498041737 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" node="3791k8s00000003" result_code="succeeded"
I0922 05:20:29.283972       1 utils.go:85] GRPC response: {}
I0922 05:20:30.687529       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 29 lines ...
I0922 05:20:43.370102       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0922 05:20:43.370365       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"}
I0922 05:20:43.370465       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0)
I0922 05:20:43.392684       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0/status 200 OK in 21 milliseconds
I0922 05:20:43.393360       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="4e3e558e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=22662710
I0922 05:20:43.393481       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-f426f608-0ffc-486a-afb4-0aca226d9da0)
I0922 05:20:43.405796       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:20:43.406853       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0 200 OK in 13 milliseconds
I0922 05:20:43.418697       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:20:43.420551       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0/status 200 OK in 11 milliseconds
I0922 05:20:43.421305       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=12550559
I0922 05:20:43.421407       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:20:43.421631       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-f426f608-0ffc-486a-afb4-0aca226d9da0)" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:20:43.421731       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-f426f608-0ffc-486a-afb4-0aca226d9da0)" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
I0922 05:20:43.421855       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-f426f608-0ffc-486a-afb4-0aca226d9da0." "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"
... skipping 5 lines ...
I0922 05:20:46.806535       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 05:20:46.817425       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 10 milliseconds
I0922 05:20:46.818061       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:20:48.589024       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.142906465 request="disks_delete" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:20:48.589055       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0
I0922 05:20:48.589114       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="fb2ebd4b-3a35-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=5166929735
I0922 05:20:48.609580       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:20:48.609801       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="4e3e558e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=5200142743
I0922 05:20:48.609852       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "disk.csi.azure.com/request-id"="4e3e558e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=5239187983
I0922 05:20:48.609894       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0) returned with <nil>
I0922 05:20:48.609982       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.239455495 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" result_code="succeeded"
I0922 05:20:48.610013       1 utils.go:85] GRPC response: {}
I0922 05:20:48.614723       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0 200 OK in 25 milliseconds
... skipping 15 lines ...
I0922 05:20:56.700942       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0922 05:20:56.701181       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-9deeb818-d0f4-4363-92c4-8f0878167279","parameters":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-9deeb818-d0f4-4363-92c4-8f0878167279","csi.storage.k8s.io/pvc/name":"pvc-j2tl4","csi.storage.k8s.io/pvc/namespace":"azuredisk-4946","maxshares":"2","skuname":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}}]}
I0922 05:20:56.701734       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:20:56.718656       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 15 milliseconds
I0922 05:20:56.724702       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:20:56.724916       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-9deeb818-d0f4-4363-92c4-8f0878167279)
I0922 05:20:56.728517       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:20:56.732895       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9deeb818-d0f4-4363-92c4-8f0878167279/status 200 OK in 14 milliseconds
I0922 05:20:56.733108       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=15137743
I0922 05:20:56.733127       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:20:56.733184       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-9deeb818-d0f4-4363-92c4-8f0878167279) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:20:56.748442       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 05:20:56.748539       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015267849 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 05:20:56.748571       1 cloudprovisioner.go:246] begin to create disk(pvc-9deeb818-d0f4-4363-92c4-8f0878167279) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone() maxShares(2)
I0922 05:20:56.809364       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.060748678 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:20:56.809417       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-9deeb818-d0f4-4363-92c4-8f0878167279 StorageAccountType:StandardSSD_ZRS Size:10
I0922 05:20:56.878700       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 05:20:56.884109       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 7 lines ...
I0922 05:20:59.043529       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-9deeb818-d0f4-4363-92c4-8f0878167279) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-9deeb818-d0f4-4363-92c4-8f0878167279 kubernetes.io-created-for-pvc-name:pvc-j2tl4 kubernetes.io-created-for-pvc-namespace:azuredisk-4946]) successfully" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:20:59.043706       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=2310511680
I0922 05:20:59.053556       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9deeb818-d0f4-4363-92c4-8f0878167279/status 200 OK in 9 milliseconds
I0922 05:20:59.055161       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=11380083
I0922 05:20:59.055465       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=2322247178
I0922 05:20:59.055756       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=2337741836
I0922 05:20:59.057739       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:20:59.057798       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=2332511614
I0922 05:20:59.057836       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=2356156717
I0922 05:20:59.057878       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.35621862 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279" result_code="succeeded"
I0922 05:20:59.057899       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-9deeb818-d0f4-4363-92c4-8f0878167279","csi.storage.k8s.io/pvc/name":"pvc-j2tl4","csi.storage.k8s.io/pvc/namespace":"azuredisk-4946","maxshares":"2","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279"}}
I0922 05:21:00.557656       1 reflector.go:536] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Watch close - *v1beta2.AzVolume total 35 items received
I0922 05:21:00.560065       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=18130&timeoutSeconds=542&watch=true 200 OK in 2 milliseconds
... skipping 15 lines ...
I0922 05:21:01.119790       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name!=3791k8s00000002,disk.csi.azure.com/volume-name=pvc-9deeb818-d0f4-4363-92c4-8f0878167279." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:21:01.138581       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 18 milliseconds
I0922 05:21:01.139797       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000002-attachment)
I0922 05:21:01.185186       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000002-attachment/status 200 OK in 58 milliseconds
I0922 05:21:01.186537       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=59799939
I0922 05:21:01.186784       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:21:01.194249       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:21:01.209453       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022399651 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:21:01.209515       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:21:01.228523       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018960605 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:21:01.228599       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:21:01.228723       1 cloudprovisioner.go:398]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279\" to node \"3791k8s00000002\"." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:21:01.228832       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279\" to node \"3791k8s00000002\"." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:21:02.229096       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-hpczfw5p|3791k8s00000002" delay="1s"
I0922 05:21:02.229481       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:21:02.247074       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017465542 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:21:02.247109       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:21:02.247161       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:21:02.259151       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:21:02.259242       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=1119242412
I0922 05:21:02.259288       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=1119527223
I0922 05:21:02.259335       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=1140042695
I0922 05:21:02.259382       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.1401733 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279" node="3791k8s00000002" result_code="succeeded"
I0922 05:21:02.259410       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 05:21:02.262130       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000002-attachment/status 200 OK in 14 milliseconds
I0922 05:21:02.262684       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=15226946
I0922 05:21:02.267292       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020006949 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:21:02.267324       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:21:02.267366       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:AttachDiskOptions{diskName: "pvc-9deeb818-d0f4-4363-92c4-8f0878167279", lun: 0}]
I0922 05:21:02.267415       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:AttachDiskOptions{diskName: "pvc-9deeb818-d0f4-4363-92c4-8f0878167279", lun: 0}])
I0922 05:21:02.448450       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.180574865 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 05:21:02.448498       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:AttachDiskOptions{diskName: "pvc-9deeb818-d0f4-4363-92c4-8f0878167279", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 05:21:02.887052       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:02.888120       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:02.888300       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:02.917835       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 05:21:02.926176       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 05:21:02.927774       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 154 lines ...
I0922 05:22:10.163489       1 shared_state.go:330]  "msg"="Requested volumes for pod azuredisk-4946/azuredisk-volume-tester-dmqtv are now the following: Volumes: [pvc-9deeb818-d0f4-4363-92c4-8f0878167279], Len: 1" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
I0922 05:22:10.163671       1 shared_state.go:588]  "msg"="Getting ranked list of nodes for creating AzVolumeAttachments" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
I0922 05:22:10.164000       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-9deeb818-d0f4-4363-92c4-8f0878167279) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
I0922 05:22:10.164204       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-9deeb818-d0f4-4363-92c4-8f0878167279)" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
I0922 05:22:10.164438       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/requested-role=Primary,disk.csi.azure.com/volume-name=pvc-9deeb818-d0f4-4363-92c4-8f0878167279." "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
I0922 05:22:10.164623       1 shared_state.go:783]  "msg"="The list of zones to select nodes from is: westus2-1,westus2-2,westus2-3" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
I0922 05:22:10.164823       1 shared_state.go:795]  "msg"="failed to find zone annotations for primary node" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
I0922 05:22:10.165221       1 common.go:394]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv" "filter-plugin"="inter-pod affinity filter" "latency"=15201
I0922 05:22:10.165320       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod affinity filter): [k8s-master-37910252-vmss000000 3791k8s00000002 3791k8s00000000]" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
I0922 05:22:10.165510       1 common.go:511]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAntiAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv" "filter-plugin"="inter-pod anti-affinity filter" "latency"=82304
I0922 05:22:10.165606       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod anti-affinity filter): [k8s-master-37910252-vmss000000 3791k8s00000002 3791k8s00000000]" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
I0922 05:22:10.165810       1 common.go:582]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*podTolerationFilter).filter" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv" "filter-plugin"="pod toleration filter" "latency"=11200
I0922 05:22:10.165892       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (pod toleration filter): [k8s-master-37910252-vmss000000 3791k8s00000002 3791k8s00000000]" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi/azure.com/pod-name"="azuredisk-4946/azuredisk-volume-tester-dmqtv"
... skipping 73 lines ...
I0922 05:22:11.282281       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=15018483
I0922 05:22:11.284949       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017827811 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:22:11.284979       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:22:11.284996       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:AttachDiskOptions{diskName: "pvc-9deeb818-d0f4-4363-92c4-8f0878167279", lun: 0}]
I0922 05:22:11.285271       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:AttachDiskOptions{diskName: "pvc-9deeb818-d0f4-4363-92c4-8f0878167279", lun: 0}])
I0922 05:22:11.443520       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.158171592 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 05:22:11.443759       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:AttachDiskOptions{diskName: "pvc-9deeb818-d0f4-4363-92c4-8f0878167279", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 05:22:11.470051       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 05:22:11.476187       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 05:22:11.476964       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:22:13.483547       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 05:22:13.492168       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 05:22:13.492663       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 81 lines ...
I0922 05:22:32.435162       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0922 05:22:32.435198       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279"}
I0922 05:22:32.435621       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279)
I0922 05:22:32.457214       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9deeb818-d0f4-4363-92c4-8f0878167279/status 200 OK in 20 milliseconds
I0922 05:22:32.458329       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="8f405d2b-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=22439212
I0922 05:22:32.458361       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-9deeb818-d0f4-4363-92c4-8f0878167279)
I0922 05:22:32.467476       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:22:32.469667       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9deeb818-d0f4-4363-92c4-8f0878167279 200 OK in 11 milliseconds
I0922 05:22:32.489706       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9deeb818-d0f4-4363-92c4-8f0878167279/status 200 OK in 22 milliseconds
I0922 05:22:32.490883       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=23980381
I0922 05:22:32.492731       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:22:32.492837       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:22:32.493538       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-9deeb818-d0f4-4363-92c4-8f0878167279)" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:22:32.493609       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-9deeb818-d0f4-4363-92c4-8f0878167279)" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:22:32.493727       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-9deeb818-d0f4-4363-92c4-8f0878167279." "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:22:32.506093       1 round_trippers.go:553] PATCH https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000002-attachment/status 200 OK in 11 milliseconds
I0922 05:22:32.522651       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000002-attachment 200 OK in 15 milliseconds
I0922 05:22:32.527815       1 shared_state.go:1028]  "msg"="Set deletion timestamp for AzVolumeAttachment (pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000002-attachment)" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:22:32.541447       1 round_trippers.go:553] PATCH https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000001-attachment/status 200 OK in 12 milliseconds
I0922 05:22:32.553313       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=59712291
I0922 05:22:32.553346       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000002-attachment)
I0922 05:22:32.553485       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000001-attachment)
I0922 05:22:32.556729       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:22:32.559972       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000002-attachment/status 200 OK in 36 milliseconds
I0922 05:22:32.561116       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=37778702
I0922 05:22:32.561320       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:22:32.561600       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279 from node 3791k8s00000002" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:22:32.561628       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:22:32.580074       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018330726 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:22:32.580318       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:22:32.888731       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:22:32.889934       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:22:32.890141       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:22:32.890316       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:22:33.477597       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:22:33.477597       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:22:33.477644       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:22:33.477941       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:22:33.478025       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:22:33.581674       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-hpczfw5p|3791k8s00000002" delay="1s"
I0922 05:22:33.581745       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:22:33.602330       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020532026 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:22:33.602493       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:22:33.602586       1 azure_controller_common.go:405] azuredisk - trying to detach disks from node 3791k8s00000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:pvc-9deeb818-d0f4-4363-92c4-8f0878167279]
I0922 05:22:33.602718       1 azure_controller_vmss.go:162] azureDisk - detach disk: name pvc-9deeb818-d0f4-4363-92c4-8f0878167279 uri /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279
... skipping 16 lines ...
I0922 05:22:43.707052       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 13 milliseconds
I0922 05:22:43.714025       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:22:43.714288       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:22:45.720583       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:22:45.729965       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0922 05:22:45.730741       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
E0922 05:22:47.435139       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="8f405d2b-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=14963751858
E0922 05:22:47.435214       1 crdprovisioner.go:368]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/request-id"="8f405d2b-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=14999335661
I0922 05:22:47.435230       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279) returned with context deadline exceeded
I0922 05:22:47.435281       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=14.999627575 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279" result_code="failed"
E0922 05:22:47.435300       1 utils.go:83] GRPC error: context deadline exceeded
I0922 05:22:47.739415       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0922 05:22:47.744333       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 05:22:47.744729       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:22:48.042625       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=20.29582238 request="vmssvm_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="detach_disk" error_code=""
I0922 05:22:48.042662       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:pvc-9deeb818-d0f4-4363-92c4-8f0878167279]) returned with <nil>
I0922 05:22:48.042701       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 3791k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:pvc-9deeb818-d0f4-4363-92c4-8f0878167279]
I0922 05:22:48.042879       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-9deeb818-d0f4-4363-92c4-8f0878167279, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279) succeeded
I0922 05:22:48.042941       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279 from node 3791k8s00000001 successfully" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:22:48.042991       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=21437623552
I0922 05:22:48.053719       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000001-attachment 200 OK in 10 milliseconds
I0922 05:22:48.054065       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=11005591
I0922 05:22:48.054132       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=21448775250
I0922 05:22:48.054173       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="81f9960a-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=21459138214
I0922 05:22:48.058053       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:22:48.058212       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4.5" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=15504467430
I0922 05:22:48.887729       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=15.28472013 request="vmssvm_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="detach_disk" error_code=""
I0922 05:22:48.887766       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000002) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:pvc-9deeb818-d0f4-4363-92c4-8f0878167279]) returned with <nil>
I0922 05:22:48.887811       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 3791k8s00000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279:pvc-9deeb818-d0f4-4363-92c4-8f0878167279]
I0922 05:22:48.887854       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-9deeb818-d0f4-4363-92c4-8f0878167279, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279) succeeded
I0922 05:22:48.887962       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279 from node 3791k8s00000002 successfully" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
I0922 05:22:48.888047       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=16326473184
I0922 05:22:48.901738       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9deeb818-d0f4-4363-92c4-8f0878167279-3791k8s00000002-attachment 200 OK in 13 milliseconds
I0922 05:22:48.902598       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=14477754
I0922 05:22:48.902909       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=16341453361
I0922 05:22:48.903169       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="58d28f6f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=16379807989
I0922 05:22:48.905434       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:22:48.905537       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4.5" "csi.storage.k8s.io/pv/name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="56307c92-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=16351885231
I0922 05:22:48.934957       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.029311823 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:22:49.750501       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 05:22:49.756944       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:22:49.757272       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:22:51.764025       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
... skipping 28 lines ...
I0922 05:23:02.890853       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:23:02.890870       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:23:03.437389       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0922 05:23:03.437418       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279"}
I0922 05:23:03.437509       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279)
I0922 05:23:03.437762       1 crdprovisioner.go:372]  "msg"="Deletion successful: could not find the AzVolume CRI" "csi.storage.k8s.io/pv/name"="" "disk.csi.azure.com/request-id"="a1badfd6-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279"
E0922 05:23:03.437838       1 crdprovisioner.go:368]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [azvolume.disk.csi.azure.com \"pvc-9deeb818-d0f4-4363-92c4-8f0878167279\" not found]" "csi.storage.k8s.io/pv/name"="" "disk.csi.azure.com/request-id"="a1badfd6-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="" "latency"=76003
I0922 05:23:03.437853       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279) returned with <nil>
I0922 05:23:03.437901       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.000359116 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-9deeb818-d0f4-4363-92c4-8f0878167279" result_code="succeeded"
I0922 05:23:03.437933       1 utils.go:85] GRPC response: {}
I0922 05:23:03.478407       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:23:03.478406       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:23:03.478423       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 19 lines ...
I0922 05:23:13.818888       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:23:13.818920       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-adb47171-c0e5-41c1-8671-e334003b87b1)
I0922 05:23:13.829810       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-adb47171-c0e5-41c1-8671-e334003b87b1/status 200 OK in 14 milliseconds
I0922 05:23:13.830392       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=15213952
I0922 05:23:13.830526       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:23:13.830838       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-adb47171-c0e5-41c1-8671-e334003b87b1) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:23:13.830859       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:23:13.870044       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 05:23:13.870398       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.039167278 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 05:23:13.870432       1 cloudprovisioner.go:246] begin to create disk(pvc-adb47171-c0e5-41c1-8671-e334003b87b1) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone() maxShares(0)
I0922 05:23:13.889351       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018871709 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:23:13.889396       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-adb47171-c0e5-41c1-8671-e334003b87b1 StorageAccountType:StandardSSD_ZRS Size:10
I0922 05:23:13.925610       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 05:23:13.934304       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
... skipping 3 lines ...
I0922 05:23:15.949352       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:23:16.143239       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.253694796 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:23:16.167388       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.024081332 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:23:16.167425       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-adb47171-c0e5-41c1-8671-e334003b87b1 StorageAccountType:StandardSSD_ZRS Size:10
I0922 05:23:16.167490       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-adb47171-c0e5-41c1-8671-e334003b87b1) account type(StandardSSD_ZRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-adb47171-c0e5-41c1-8671-e334003b87b1 kubernetes.io-created-for-pvc-name:pvc-9twjm kubernetes.io-created-for-pvc-namespace:azuredisk-9888]) successfully" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:23:16.167582       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=2336789557
I0922 05:23:16.183176       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:23:16.183260       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=2364105828
I0922 05:23:16.183305       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=2382592420
I0922 05:23:16.183592       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.383004638 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1" result_code="succeeded"
I0922 05:23:16.183649       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-adb47171-c0e5-41c1-8671-e334003b87b1","csi.storage.k8s.io/pvc/name":"pvc-9twjm","csi.storage.k8s.io/pvc/namespace":"azuredisk-9888","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1"}}
I0922 05:23:16.186003       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-adb47171-c0e5-41c1-8671-e334003b87b1/status 200 OK in 18 milliseconds
I0922 05:23:16.187354       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=19627041
... skipping 12 lines ...
I0922 05:23:18.338368       1 shared_state.go:483]  "msg"="Storing pod azuredisk-volume-tester-9frgj-79b66f99b6-j9m8k and claim [] to podToClaimsMap map." 
I0922 05:23:18.415673       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0922 05:23:18.415890       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-adb47171-c0e5-41c1-8671-e334003b87b1","csi.storage.k8s.io/pvc/name":"pvc-9twjm","csi.storage.k8s.io/pvc/namespace":"azuredisk-9888","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1"}
I0922 05:23:18.416287       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=3791k8s00000001,disk.csi.azure.com/volume-name!=pvc-adb47171-c0e5-41c1-8671-e334003b87b1." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:23:18.421888       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 5 milliseconds
I0922 05:23:18.422342       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-adb47171-c0e5-41c1-8671-e334003b87b1-3791k8s00000001-attachment)
I0922 05:23:18.425268       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:23:18.435319       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-adb47171-c0e5-41c1-8671-e334003b87b1-3791k8s00000001-attachment/status 200 OK in 9 milliseconds
I0922 05:23:18.439435       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:23:18.440472       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=15057045
I0922 05:23:18.440687       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:23:18.464929       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023853322 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:23:18.464990       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:23:18.482859       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017825964 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:23:18.482904       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
... skipping 9 lines ...
I0922 05:23:19.632761       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-adb47171-c0e5-41c1-8671-e334003b87b1-3791k8s00000001-attachment/status 200 OK in 13 milliseconds
I0922 05:23:19.635672       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016729717 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:23:19.635797       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:23:19.635908       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1:AttachDiskOptions{diskName: "pvc-adb47171-c0e5-41c1-8671-e334003b87b1", lun: 0}]
I0922 05:23:19.636083       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1:AttachDiskOptions{diskName: "pvc-adb47171-c0e5-41c1-8671-e334003b87b1", lun: 0}])
I0922 05:23:19.637410       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=18271283
I0922 05:23:19.637448       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:23:19.637883       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=1215422694
I0922 05:23:19.638136       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=1215835112
I0922 05:23:19.638433       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=1222282989
I0922 05:23:19.638680       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.2225722 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1" node="3791k8s00000001" result_code="succeeded"
I0922 05:23:19.638910       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 05:23:19.760424       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.124189886 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 05:23:19.760465       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1:AttachDiskOptions{diskName: "pvc-adb47171-c0e5-41c1-8671-e334003b87b1", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 05:23:19.968331       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 05:23:19.976906       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 05:23:19.977611       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:23:21.987916       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0922 05:23:21.994302       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 05:23:21.994600       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 175 lines ...
I0922 05:24:32.892162       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:24:32.892163       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:24:32.893272       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:24:33.480615       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:24:33.480876       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:24:33.481039       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:24:33.918602       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="3791k8s00000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0922 05:24:33.919071       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d7a92b1f-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=18501
I0922 05:24:34.518584       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 05:24:34.528695       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:24:34.529146       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:24:36.537581       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 05:24:36.543107       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 22 lines ...
I0922 05:24:44.088963       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-adb47171-c0e5-41c1-8671-e334003b87b1-3791k8s00000001-attachment)
I0922 05:24:44.093572       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-adb47171-c0e5-41c1-8671-e334003b87b1-3791k8s00000001-attachment/status 200 OK in 8 milliseconds
I0922 05:24:44.094234       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=9072989
I0922 05:24:44.094277       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:24:44.094562       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1 from node 3791k8s00000001" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:24:44.094715       1 azure_vmss_cache.go:332] Node 3791k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:24:44.098389       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:24:44.112955       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018192979 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:24:44.112982       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:24:44.113113       1 azure_vmss.go:204] Couldn't find VMSS VM with nodeName 3791k8s00000001, refreshing the cache(vmss: 3791k8s00, rg: kubetest-hpczfw5p)
I0922 05:24:44.223694       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.110529438 request="vmssvm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:24:44.590495       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0922 05:24:44.597454       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
... skipping 30 lines ...
I0922 05:25:00.591681       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1:pvc-adb47171-c0e5-41c1-8671-e334003b87b1]) returned with <nil>
I0922 05:25:00.591718       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 3791k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1:pvc-adb47171-c0e5-41c1-8671-e334003b87b1]
I0922 05:25:00.591771       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-adb47171-c0e5-41c1-8671-e334003b87b1, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1) succeeded
I0922 05:25:00.591823       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1 from node 3791k8s00000001 successfully" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:25:00.591890       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=16497339080
I0922 05:25:00.602557       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-adb47171-c0e5-41c1-8671-e334003b87b1-3791k8s00000001-attachment 200 OK in 10 milliseconds
I0922 05:25:00.605904       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:25:00.605989       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=16516859039
I0922 05:25:00.606052       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=16517126151
I0922 05:25:00.606106       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="ddb56524-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=16540974073
I0922 05:25:00.606179       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.541255285 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1" node="3791k8s00000001" result_code="succeeded"
I0922 05:25:00.606197       1 utils.go:85] GRPC response: {}
I0922 05:25:00.606570       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="3791k8s00000001" "disk.csi.azure.com/request-id"="aaa8657d-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=14624350
... skipping 29 lines ...
I0922 05:25:09.227652       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1"}
I0922 05:25:09.227988       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1)
I0922 05:25:09.243750       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-adb47171-c0e5-41c1-8671-e334003b87b1/status 200 OK in 15 milliseconds
I0922 05:25:09.249711       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="ecb4ff19-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=21339447
I0922 05:25:09.249740       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-adb47171-c0e5-41c1-8671-e334003b87b1)
I0922 05:25:09.259365       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-adb47171-c0e5-41c1-8671-e334003b87b1 200 OK in 9 milliseconds
I0922 05:25:09.263707       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:25:09.274421       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:25:09.274719       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-adb47171-c0e5-41c1-8671-e334003b87b1/status 200 OK in 11 milliseconds
I0922 05:25:09.277313       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=15102470
I0922 05:25:09.277586       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:25:09.277734       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-adb47171-c0e5-41c1-8671-e334003b87b1)" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:25:09.277784       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-adb47171-c0e5-41c1-8671-e334003b87b1)" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
I0922 05:25:09.277845       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-adb47171-c0e5-41c1-8671-e334003b87b1." "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1"
... skipping 9 lines ...
I0922 05:25:14.485891       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1
I0922 05:25:14.486132       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=5207941543
I0922 05:25:14.498670       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-adb47171-c0e5-41c1-8671-e334003b87b1 200 OK in 12 milliseconds
I0922 05:25:14.499627       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=13418895
I0922 05:25:14.499714       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=5222026368
I0922 05:25:14.499863       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="a7e82361-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=5237634959
I0922 05:25:14.503468       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:25:14.504222       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="ecb4ff19-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=5243335313
I0922 05:25:14.504327       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "disk.csi.azure.com/request-id"="ecb4ff19-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-adb47171-c0e5-41c1-8671-e334003b87b1" "latency"=5275934058
I0922 05:25:14.504344       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1) returned with <nil>
I0922 05:25:14.504537       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.276529984 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-adb47171-c0e5-41c1-8671-e334003b87b1" result_code="succeeded"
I0922 05:25:14.504716       1 utils.go:85] GRPC response: {}
I0922 05:25:14.835173       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 25 milliseconds
... skipping 20 lines ...
I0922 05:25:23.479414       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0922 05:25:23.479873       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-495b02b3-0776-498a-b8a3-8c2434c639bf","parameters":{"csi.storage.k8s.io/pv/name":"pvc-495b02b3-0776-498a-b8a3-8c2434c639bf","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0922 05:25:23.481248       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "disk.csi.azure.com/request-id"="f533cd18-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf"
I0922 05:25:23.497153       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 15 milliseconds
I0922 05:25:23.498872       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "disk.csi.azure.com/request-id"="f533cd18-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf"
I0922 05:25:23.498910       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-495b02b3-0776-498a-b8a3-8c2434c639bf)
I0922 05:25:23.501482       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:25:23.509596       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf/status 200 OK in 6 milliseconds
I0922 05:25:23.510039       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "disk.csi.azure.com/request-id"="f533cd18-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=7362826
I0922 05:25:23.510077       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "disk.csi.azure.com/request-id"="f533cd18-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf"
I0922 05:25:23.512408       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:25:23.525460       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 05:25:23.525679       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015222173 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 05:25:23.525712       1 cloudprovisioner.go:246] begin to create disk(pvc-495b02b3-0776-498a-b8a3-8c2434c639bf) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0922 05:25:23.555073       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02917589 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:25:23.555112       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-495b02b3-0776-498a-b8a3-8c2434c639bf StorageAccountType:StandardSSD_LRS Size:10
I0922 05:25:24.915387       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:25:24.920812       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:25:24.921389       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:25:25.765493       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.210316786 request="disks_create_or_update" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:25:25.786273       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020705116 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:25:25.786447       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-495b02b3-0776-498a-b8a3-8c2434c639bf StorageAccountType:StandardSSD_LRS Size:10
I0922 05:25:25.786614       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-495b02b3-0776-498a-b8a3-8c2434c639bf) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-495b02b3-0776-498a-b8a3-8c2434c639bf kubernetes.io-created-for-pvc-name:pvc-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "disk.csi.azure.com/request-id"="f533cd18-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf"
I0922 05:25:25.786769       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "disk.csi.azure.com/request-id"="f533cd18-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=2276466612
I0922 05:25:25.798595       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:25:25.799593       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "disk.csi.azure.com/request-id"="f533cd18-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=2300069756
I0922 05:25:25.799688       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "disk.csi.azure.com/request-id"="f533cd18-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=2318508173
I0922 05:25:25.799847       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.318699481 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" result_code="succeeded"
I0922 05:25:25.799889       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-495b02b3-0776-498a-b8a3-8c2434c639bf","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf"}}
I0922 05:25:25.813389       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf/status 200 OK in 26 milliseconds
I0922 05:25:25.822337       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "disk.csi.azure.com/request-id"="f533cd18-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=35432068
... skipping 2 lines ...
I0922 05:25:26.564862       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0922 05:25:26.565059       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000003","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-495b02b3-0776-498a-b8a3-8c2434c639bf","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf"}
I0922 05:25:26.565473       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=3791k8s00000003,disk.csi.azure.com/volume-name!=pvc-495b02b3-0776-498a-b8a3-8c2434c639bf." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="f70a6a6e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf"
I0922 05:25:26.572511       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 6 milliseconds
I0922 05:25:26.573554       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-495b02b3-0776-498a-b8a3-8c2434c639bf-3791k8s00000003-attachment)
I0922 05:25:26.581040       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf-3791k8s00000003-attachment/status 200 OK in 6 milliseconds
I0922 05:25:26.582781       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:25:26.583480       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="f70a6a6e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=9086502
I0922 05:25:26.584360       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="f70a6a6e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf"
I0922 05:25:26.607286       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022674903 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:25:26.607349       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:25:26.625749       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018346611 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:25:26.625777       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
... skipping 7 lines ...
I0922 05:25:27.689620       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:25:27.707301       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017630279 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:25:27.707333       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:25:27.707429       1 azure_vmss_cache.go:332] Node 3791k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:25:27.718462       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf-3791k8s00000003-attachment/status 200 OK in 10 milliseconds
I0922 05:25:27.719539       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="f70a6a6e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=11949428
I0922 05:25:27.721294       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:25:27.721548       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="f70a6a6e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=1147874983
I0922 05:25:27.721692       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="f70a6a6e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=1148069391
I0922 05:25:27.721884       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="f70a6a6e-3a36-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=1156444262
I0922 05:25:27.721992       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.156716874 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" node="3791k8s00000003" result_code="succeeded"
I0922 05:25:27.722072       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 05:25:27.765084       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.057419941 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:25:27.765115       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:25:27.765131       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf:AttachDiskOptions{diskName: "pvc-495b02b3-0776-498a-b8a3-8c2434c639bf", lun: 0}]
I0922 05:25:27.765280       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf:AttachDiskOptions{diskName: "pvc-495b02b3-0776-498a-b8a3-8c2434c639bf", lun: 0}])
I0922 05:25:27.889924       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.124441906 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 05:25:27.889959       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf:AttachDiskOptions{diskName: "pvc-495b02b3-0776-498a-b8a3-8c2434c639bf", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 05:25:28.946164       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0922 05:25:28.951255       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 05:25:28.951929       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:25:30.958203       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0922 05:25:30.968894       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0922 05:25:30.970043       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 103 lines ...
I0922 05:26:05.997378       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 19 milliseconds
I0922 05:26:05.998825       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"
I0922 05:26:05.998935       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-127c0534-1fab-4e43-827d-215d19c9d5bd)
I0922 05:26:06.006030       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd/status 200 OK in 7 milliseconds
I0922 05:26:06.006682       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=8570078
I0922 05:26:06.006834       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"
I0922 05:26:06.009011       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:26:06.023415       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd' under resource group 'kubetest-hpczfw5p' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0922 05:26:06.023673       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016402925 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0922 05:26:06.023715       1 cloudprovisioner.go:246] begin to create disk(pvc-127c0534-1fab-4e43-827d-215d19c9d5bd) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(100) selectedAvailabilityZone(westus2-1) maxShares(0)
I0922 05:26:06.075151       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.05138977 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:26:06.075197       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-127c0534-1fab-4e43-827d-215d19c9d5bd StorageAccountType:StandardSSD_LRS Size:100
I0922 05:26:07.269372       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0922 05:26:07.275701       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
... skipping 4 lines ...
I0922 05:26:08.292028       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-127c0534-1fab-4e43-827d-215d19c9d5bd) account type(StandardSSD_LRS) rg(kubetest-hpczfw5p) location() size(100) tags(map[kubernetes.io-created-for-pv-name:pvc-127c0534-1fab-4e43-827d-215d19c9d5bd kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-win-0 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"
I0922 05:26:08.292118       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=2285058114
I0922 05:26:08.305197       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd/status 200 OK in 12 milliseconds
I0922 05:26:08.306433       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=14218928
I0922 05:26:08.306743       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=2299692261
I0922 05:26:08.306977       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=2308908267
I0922 05:26:08.307830       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:26:08.307961       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=2308849865
I0922 05:26:08.308150       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "disk.csi.azure.com/request-id"="0e882915-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=2331104848
I0922 05:26:08.308212       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.331228154 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" result_code="succeeded"
I0922 05:26:08.308272       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}}],"capacity_bytes":107374182400,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-127c0534-1fab-4e43-827d-215d19c9d5bd","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"}}
I0922 05:26:09.076097       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0922 05:26:09.076141       1 utils.go:79] GRPC request: {"node_id":"3791k8s00000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-127c0534-1fab-4e43-827d-215d19c9d5bd","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"}
I0922 05:26:09.076670       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=3791k8s00000002,disk.csi.azure.com/volume-name!=pvc-127c0534-1fab-4e43-827d-215d19c9d5bd." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="10611aa2-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"
I0922 05:26:09.082209       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 5 milliseconds
I0922 05:26:09.084105       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-127c0534-1fab-4e43-827d-215d19c9d5bd-3791k8s00000002-attachment)
I0922 05:26:09.086433       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:26:09.094733       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd-3791k8s00000002-attachment/status 200 OK in 7 milliseconds
I0922 05:26:09.095307       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="10611aa2-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=8377071
I0922 05:26:09.095622       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="10611aa2-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"
I0922 05:26:09.101334       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:26:09.128616       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.032645745 request="disks_get" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:26:09.128710       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:26:09.178125       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.049316383 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:26:09.178390       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:26:09.178687       1 cloudprovisioner.go:398]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd\" to node \"3791k8s00000002\"." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="10611aa2-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"
I0922 05:26:09.178984       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd\" to node \"3791k8s00000002\"." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="10611aa2-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"
... skipping 4 lines ...
I0922 05:26:10.179721       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:26:10.198733       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018803932 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:26:10.198762       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:26:10.199218       1 azure_vmss_cache.go:332] Node 3791k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0922 05:26:10.212215       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd-3791k8s00000002-attachment/status 200 OK in 12 milliseconds
I0922 05:26:10.213205       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="10611aa2-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=13655804
I0922 05:26:10.215551       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:26:10.217388       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="10611aa2-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=1131172363
I0922 05:26:10.217724       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="10611aa2-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=1133431864
I0922 05:26:10.217823       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="10611aa2-3a37-11ed-a99d-00224878ec48" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=1141297412
I0922 05:26:10.218003       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.141458519 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" node="3791k8s00000002" result_code="succeeded"
I0922 05:26:10.218171       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0922 05:26:10.226893       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.027588821 request="vm_list" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0922 05:26:10.226921       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-hpczfw5p) success
I0922 05:26:10.226938       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 3791k8s00000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd:AttachDiskOptions{diskName: "pvc-127c0534-1fab-4e43-827d-215d19c9d5bd", lun: 0}]
I0922 05:26:10.226981       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd:AttachDiskOptions{diskName: "pvc-127c0534-1fab-4e43-827d-215d19c9d5bd", lun: 0}])
I0922 05:26:10.375891       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.148842188 request="vmssvm_updateasync" resource_group="kubetest-hpczfw5p" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0922 05:26:10.375938       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-hpczfw5p): vm(3791k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd:AttachDiskOptions{diskName: "pvc-127c0534-1fab-4e43-827d-215d19c9d5bd", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0922 05:26:11.292894       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0922 05:26:11.298006       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0922 05:26:11.299099       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0922 05:26:13.309226       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0922 05:26:13.316122       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0922 05:26:13.316348       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 145 lines ...
I0922 04:21:05.411981    5912 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=1751&timeout=5m59s&timeoutSeconds=359&watch=true 200 OK in 2 milliseconds
I0922 04:21:05.411981    5912 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1512&timeout=8m31s&timeoutSeconds=511&watch=true 200 OK in 3 milliseconds
I0922 04:21:05.513228    5912 shared_informer.go:285] caches populated
I0922 04:21:05.513289    5912 azuredisk_v2.go:193] driver userAgent: disk.csi.azure.com/latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad e2e-test
I0922 04:21:05.513870    5912 azure_disk_utils.go:543] reading cloud config from secret kube-system/azure-cloud-provider
I0922 04:21:05.520005    5912 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 4 milliseconds
I0922 04:21:05.521530    5912 azure_disk_utils.go:550] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0922 04:21:05.521580    5912 azure_disk_utils.go:555] could not read cloud config from secret kube-system/azure-cloud-provider
I0922 04:21:05.521580    5912 azure_disk_utils.go:565] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0922 04:21:05.530811    5912 azure_disk_utils.go:573] read cloud config from file: C:\k\azure.json successfully
I0922 04:21:05.533294    5912 azure_auth.go:245] Using AzurePublicCloud environment
I0922 04:21:05.533875    5912 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I0922 04:21:05.533875    5912 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
I0922 04:21:05.533875    5912 azure_auth.go:113] azure: User Assigned MSI ID is client ID. Resource ID parsing error: %+vparsing failed for d55c9a9f-4cdb-4884-bfff-b50f401b2560. Invalid resource Id format
I0922 04:21:05.538882    5912 azure_diskclient.go:68] Azure DisksClient using API version: 2021-04-01
I0922 04:21:05.542891    5912 azure.go:1011] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333333, Bucket: 80
I0922 04:21:05.547362    5912 safe_mounter_windows.go:406] using CSIProxyMounterV1, API Versions filesystem: v1, disk: v1, volume: v1
I0922 04:21:05.548112    5912 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0922 04:21:05.548112    5912 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0922 04:21:05.548112    5912 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
... skipping 426 lines ...
I0922 04:39:02.729823    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:39:02.729823    5912 utils.go:79] GRPC request: {}
I0922 04:39:02.729923    5912 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 04:39:05.765638    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:39:05.765638    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:39:05.765638    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:39:05.765638    5912 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:39:05.765638    5912 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:39:05.935295    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:39:06.426983    5912 azuredisk_v2.go:551]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-22T04:39:06Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="7df2f80d-3a30-11ed-9e26-00155d698167"
I0922 04:39:06.435172    5912 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/3791k8s00000002/status 200 OK in 7 milliseconds
I0922 04:39:06.435876    5912 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="7df2f80d-3a30-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8892300
I0922 04:39:19.759412    5912 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:39:19.759412    5912 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="70cb3489-3a30-11ed-9e25-00155d698167" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0" "latency"=35401789900
I0922 04:39:19.759412    5912 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="70cb3489-3a30-11ed-9e25-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35401857700
I0922 04:39:19.760359    5912 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:39:19.760493    5912 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="70cb3489-3a30-11ed-9e26-00155d698167" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-19963529-e901-485c-9886-a19898f82d56" "latency"=35402737300
I0922 04:39:19.760561    5912 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="70cb3489-3a30-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=35402939400
I0922 04:39:25.682156    5912 nodeserver_v2.go:211] NodeStageVolume: formatting 7 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-40426928-1c99-46a9-92f3-0f6ea80e8ab0\globalmount with mount options([])
I0922 04:39:25.685517    5912 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-19963529-e901-485c-9886-a19898f82d56\globalmount with mount options([])
I0922 04:39:32.742567    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:39:32.742567    5912 utils.go:79] GRPC request: {}
... skipping 202 lines ...
I0922 04:45:02.758318    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:45:02.758318    5912 utils.go:79] GRPC request: {}
I0922 04:45:02.758403    5912 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 04:45:05.868006    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:45:05.868006    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:45:05.868317    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:45:05.868435    5912 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:45:06.060330    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:45:06.448846    5912 azuredisk_v2.go:551]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-22T04:45:06Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="5489f215-3a31-11ed-9e26-00155d698167"
I0922 04:45:06.457587    5912 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/3791k8s00000002/status 200 OK in 7 milliseconds
I0922 04:45:06.458787    5912 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="5489f215-3a31-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=9940300
I0922 04:45:15.809789    5912 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:45:15.809789    5912 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="51edc793-3a31-11ed-9e26-00155d698167" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-0bcd181b-5d02-4a22-9f84-f0209e876bae" "latency"=13739072400
I0922 04:45:15.809789    5912 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="51edc793-3a31-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=13739142300
I0922 04:45:20.243745    5912 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-0bcd181b-5d02-4a22-9f84-f0209e876bae\globalmount with mount options([barrier acl])
I0922 04:45:32.757952    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:45:32.757952    5912 utils.go:79] GRPC request: {}
I0922 04:45:32.757952    5912 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 193 lines ...
I0922 04:52:32.790912    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:52:32.790912    5912 utils.go:79] GRPC request: {}
I0922 04:52:32.791079    5912 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 04:52:36.014035    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:52:36.014035    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:52:36.014293    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:52:36.014356    5912 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:52:36.186786    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:52:36.473823    5912 azuredisk_v2.go:551]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-22T04:52:36Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="60c6456a-3a32-11ed-9e26-00155d698167"
I0922 04:52:36.508115    5912 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/3791k8s00000002/status 200 OK in 34 milliseconds
I0922 04:52:36.508115    5912 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="60c6456a-3a32-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=34527800
I0922 04:52:55.973816    5912 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:52:55.973953    5912 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="57c5875f-3a32-11ed-9e26-00155d698167" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9c355c2c-8bb3-4463-9031-96824e50d877" "latency"=34602711000
I0922 04:52:55.973953    5912 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="57c5875f-3a32-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=34602848500
I0922 04:53:00.332026    5912 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-9c355c2c-8bb3-4463-9031-96824e50d877\globalmount with mount options([])
I0922 04:53:02.794641    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:53:02.794641    5912 utils.go:79] GRPC request: {}
I0922 04:53:02.794691    5912 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 70 lines ...
I0922 04:54:33.320076    5912 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 04:54:33.320076    5912 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-2bd80e2b-4e77-417e-8603-fa2309d62668\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2bd80e2b-4e77-417e-8603-fa2309d62668","csi.storage.k8s.io/pvc/name":"pvc-5sjpd","csi.storage.k8s.io/pvc/namespace":"azuredisk-9971","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-2bd80e2b-4e77-417e-8603-fa2309d62668"}
I0922 04:54:33.320076    5912 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2bd80e2b-4e77-417e-8603-fa2309d62668-3791k8s00000002-attachment)
I0922 04:54:36.064298    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:54:36.064298    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:54:36.064298    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:54:36.064298    5912 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:54:36.247549    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:54:36.471673    5912 azuredisk_v2.go:551]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-22T04:54:36Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="a84c86c0-3a32-11ed-9e26-00155d698167"
I0922 04:54:36.476450    5912 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/3791k8s00000002/status 200 OK in 3 milliseconds
I0922 04:54:36.480040    5912 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="a84c86c0-3a32-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=8246000
I0922 04:54:37.468907    5912 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:54:37.469035    5912 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="a66ba178-3a32-11ed-9e26-00155d698167" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2bd80e2b-4e77-417e-8603-fa2309d62668" "latency"=4148652600
I0922 04:54:37.469035    5912 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="a66ba178-3a32-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4148694000
I0922 04:54:41.836825    5912 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2bd80e2b-4e77-417e-8603-fa2309d62668\globalmount with mount options([])
I0922 04:54:50.400270    5912 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 34 items received
I0922 04:54:50.401865    5912 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=10974&timeout=6m57s&timeoutSeconds=417&watch=true 200 OK in 0 milliseconds
I0922 04:55:02.790205    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
... skipping 613 lines ...
I0922 05:21:06.558321    5912 azuredisk_v2.go:551]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-22T05:21:06Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="5c10671b-3a36-11ed-9e26-00155d698167"
I0922 05:21:06.564118    5912 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/3791k8s00000002/status 200 OK in 4 milliseconds
I0922 05:21:06.570104    5912 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="5c10671b-3a36-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12380200
I0922 05:21:06.611823    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:06.611823    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:06.611823    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:06.612106    5912 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:21:06.770120    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:32.878362    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 05:21:32.878362    5912 utils.go:79] GRPC request: {}
I0922 05:21:32.878362    5912 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 05:21:35.444327    5912 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 36 items received
I0922 05:21:35.445851    5912 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=18284&timeout=9m58s&timeoutSeconds=598&watch=true 200 OK in 0 milliseconds
I0922 05:21:36.565542    5912 azuredisk_v2.go:551]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-22T05:21:36Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6df33b82-3a36-11ed-9e26-00155d698167"
I0922 05:21:36.577544    5912 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/3791k8s00000002/status 200 OK in 5 milliseconds
I0922 05:21:36.578676    5912 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="6df33b82-3a36-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=13133200
I0922 05:21:36.619954    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:36.619954    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:36.619954    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:36.619954    5912 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:21:36.775802    5912 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:21:37.763452    5912 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:21:37.763592    5912 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="5c0925c1-3a36-11ed-9e26-00155d698167" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9deeb818-d0f4-4363-92c4-8f0878167279" "latency"=31251287100
I0922 05:21:37.763592    5912 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="5c0925c1-3a36-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31251972100
I0922 05:21:42.741753    5912 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-9deeb818-d0f4-4363-92c4-8f0878167279\globalmount with mount options([barrier=1 acl])
I0922 05:22:02.872531    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 05:22:02.872531    5912 utils.go:79] GRPC request: {}
I0922 05:22:02.872531    5912 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 130 lines ...
I0922 05:26:11.698142    5912 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 05:26:11.702231    5912 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 05:26:11.702231    5912 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-127c0534-1fab-4e43-827d-215d19c9d5bd\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-127c0534-1fab-4e43-827d-215d19c9d5bd","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-127c0534-1fab-4e43-827d-215d19c9d5bd"}
I0922 05:26:11.702937    5912 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-127c0534-1fab-4e43-827d-215d19c9d5bd-3791k8s00000002-attachment)
I0922 05:26:15.468285    5912 reflector.go:536] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzDriverNode total 72 items received
I0922 05:26:15.473548    5912 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=19623&timeout=5m10s&timeoutSeconds=310&watch=true 200 OK in 5 milliseconds
I0922 05:26:20.520623    5912 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:26:20.520913    5912 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000002" "disk.csi.azure.com/request-id"="11f1de03-3a37-11ed-9e26-00155d698167" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-127c0534-1fab-4e43-827d-215d19c9d5bd" "latency"=8817364600
I0922 05:26:20.520913    5912 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="11f1de03-3a37-11ed-9e26-00155d698167" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=8817364600
I0922 05:26:25.327917    5912 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-127c0534-1fab-4e43-827d-215d19c9d5bd\globalmount with mount options([])
I0922 05:26:32.894095    5912 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 05:26:32.894095    5912 utils.go:79] GRPC request: {}
I0922 05:26:32.894226    5912 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 66 lines ...
I0922 04:20:40.553149    6408 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=1722&timeout=5m43s&timeoutSeconds=343&watch=true 200 OK in 5 milliseconds
I0922 04:20:40.553149    6408 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1512&timeout=5m21s&timeoutSeconds=321&watch=true 200 OK in 1 milliseconds
I0922 04:20:40.657894    6408 shared_informer.go:285] caches populated
I0922 04:20:40.657894    6408 azuredisk_v2.go:193] driver userAgent: disk.csi.azure.com/latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad e2e-test
I0922 04:20:40.658614    6408 azure_disk_utils.go:543] reading cloud config from secret kube-system/azure-cloud-provider
I0922 04:20:40.664822    6408 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 4 milliseconds
I0922 04:20:40.666006    6408 azure_disk_utils.go:550] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0922 04:20:40.666006    6408 azure_disk_utils.go:555] could not read cloud config from secret kube-system/azure-cloud-provider
I0922 04:20:40.666047    6408 azure_disk_utils.go:565] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0922 04:20:40.673628    6408 azure_disk_utils.go:573] read cloud config from file: C:\k\azure.json successfully
I0922 04:20:40.678769    6408 azure_auth.go:245] Using AzurePublicCloud environment
I0922 04:20:40.680329    6408 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I0922 04:20:40.680329    6408 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
I0922 04:20:40.680329    6408 azure_auth.go:113] azure: User Assigned MSI ID is client ID. Resource ID parsing error: %+vparsing failed for d55c9a9f-4cdb-4884-bfff-b50f401b2560. Invalid resource Id format
I0922 04:20:40.685286    6408 azure_diskclient.go:68] Azure DisksClient using API version: 2021-04-01
I0922 04:20:40.691065    6408 azure.go:1011] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333333, Bucket: 80
I0922 04:20:40.694681    6408 safe_mounter_windows.go:406] using CSIProxyMounterV1, API Versions filesystem: v1, disk: v1, volume: v1
I0922 04:20:40.694718    6408 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0922 04:20:40.694718    6408 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0922 04:20:40.694718    6408 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
... skipping 60 lines ...
I0922 04:21:48.987327    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0922 04:21:48.987364    6408 utils.go:79] GRPC request: {}
I0922 04:21:48.987477    6408 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 04:21:48.988843    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 04:21:48.988843    6408 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-b2afb6b1-3270-4daa-b997-87a148097c19\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-b2afb6b1-3270-4daa-b997-87a148097c19","csi.storage.k8s.io/pvc/name":"pvc-jgfgf","csi.storage.k8s.io/pvc/namespace":"azuredisk-824","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-b2afb6b1-3270-4daa-b997-87a148097c19"}
I0922 04:21:48.990067    6408 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-b2afb6b1-3270-4daa-b997-87a148097c19-3791k8s00000003-attachment)
I0922 04:21:58.603495    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:21:58.604141    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="139692bd-3a2e-11ed-bdff-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-b2afb6b1-3270-4daa-b997-87a148097c19" "latency"=9613287000
I0922 04:21:58.604261    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="139692bd-3a2e-11ed-bdff-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=9614610200
I0922 04:22:01.113185    6408 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:22:01.113185    6408 utils.go:79] GRPC request: {}
I0922 04:22:01.113185    6408 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 04:22:03.107140    6408 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-b2afb6b1-3270-4daa-b997-87a148097c19\globalmount with mount options([])
... skipping 409 lines ...
I0922 04:38:53.072133    6408 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50-3791k8s00000003-attachment)
I0922 04:39:01.132617    6408 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:39:01.132617    6408 utils.go:79] GRPC request: {}
I0922 04:39:01.132617    6408 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 04:39:10.869735    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:39:10.869735    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:39:10.869735    6408 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:39:10.869735    6408 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 04:39:10.968906    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:39:11.032209    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 04:39:13.685989    6408 azuredisk_v2.go:551]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-22T04:39:13Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="82469798-3a30-11ed-be00-00155d071c25"
I0922 04:39:13.701895    6408 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/3791k8s00000003/status 200 OK in 12 milliseconds
I0922 04:39:13.703375    6408 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="82469798-3a30-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=17472000
I0922 04:39:20.434852    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:39:20.435372    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="75fd2e35-3a30-11ed-be00-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50" "latency"=27362875200
I0922 04:39:20.435515    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="75fd2e35-3a30-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=27362875200
I0922 04:39:20.445809    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:39:20.447383    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="75fd2e35-3a30-11ed-bdff-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4677ae46-f78d-48a3-9eed-d62d49691b17" "latency"=27374850600
I0922 04:39:20.447383    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="75fd2e35-3a30-11ed-bdff-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=27374886200
I0922 04:39:24.338148    6408 nodeserver_v2.go:211] NodeStageVolume: formatting 7 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-50f30071-f4a0-41f7-af27-23a17a1c7c50\globalmount with mount options([])
I0922 04:39:24.409870    6408 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-4677ae46-f78d-48a3-9eed-d62d49691b17\globalmount with mount options([])
I0922 04:39:31.140147    6408 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:39:31.140147    6408 utils.go:79] GRPC request: {}
... skipping 320 lines ...
I0922 04:46:54.115158    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0922 04:46:54.115158    6408 utils.go:79] GRPC request: {}
I0922 04:46:54.115158    6408 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 04:46:54.116785    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 04:46:54.116828    6408 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-ab64bbd8-ea9a-4549-8ea3-9e9de108b334\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ab64bbd8-ea9a-4549-8ea3-9e9de108b334","csi.storage.k8s.io/pvc/name":"pvc-j5nsp","csi.storage.k8s.io/pvc/namespace":"azuredisk-4340","enableAsyncAttach":"false","enableasyncattach":"false","networkAccessPolicy":"DenyAll","requestedsizegib":"10","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ab64bbd8-ea9a-4549-8ea3-9e9de108b334"}
I0922 04:46:54.116828    6408 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ab64bbd8-ea9a-4549-8ea3-9e9de108b334-3791k8s00000003-attachment)
I0922 04:46:56.745948    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:46:56.746098    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="94b6c69d-3a31-11ed-be00-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ab64bbd8-ea9a-4549-8ea3-9e9de108b334" "latency"=2629065600
I0922 04:46:56.746098    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="94b6c69d-3a31-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2629215400
I0922 04:47:00.174978    6408 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-ab64bbd8-ea9a-4549-8ea3-9e9de108b334\globalmount with mount options([])
I0922 04:47:01.140033    6408 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:47:01.140033    6408 utils.go:79] GRPC request: {}
I0922 04:47:01.140033    6408 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 214 lines ...
I0922 04:55:23.869497    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0922 04:55:23.869497    6408 utils.go:79] GRPC request: {}
I0922 04:55:23.869497    6408 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 04:55:23.871720    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 04:55:23.871720    6408 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-ba7a35cd-e3c7-4741-af1d-9d857b546d95\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ba7a35cd-e3c7-4741-af1d-9d857b546d95","csi.storage.k8s.io/pvc/name":"pvc-r5b5g","csi.storage.k8s.io/pvc/namespace":"azuredisk-9971","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-ba7a35cd-e3c7-4741-af1d-9d857b546d95"}
I0922 04:55:23.871720    6408 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ba7a35cd-e3c7-4741-af1d-9d857b546d95-3791k8s00000003-attachment)
I0922 04:55:26.037449    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 04:55:26.037574    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="c48d3311-3a32-11ed-be00-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ba7a35cd-e3c7-4741-af1d-9d857b546d95" "latency"=2165677700
I0922 04:55:26.037574    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="c48d3311-3a32-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2165802300
I0922 04:55:29.456400    6408 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-ba7a35cd-e3c7-4741-af1d-9d857b546d95\globalmount with mount options([])
I0922 04:55:31.144854    6408 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 04:55:31.144854    6408 utils.go:79] GRPC request: {}
I0922 04:55:31.145427    6408 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 266 lines ...
I0922 05:06:09.124583    6408 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 05:06:09.125923    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 05:06:09.125923    6408 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f76bc538-dc46-46e1-a42d-fc5c0ee737b6\\globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"btrfs"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f76bc538-dc46-46e1-a42d-fc5c0ee737b6","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-t2qwb-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-7650","fsType":"btrfs","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f76bc538-dc46-46e1-a42d-fc5c0ee737b6"}
I0922 05:06:09.125923    6408 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-f76bc538-dc46-46e1-a42d-fc5c0ee737b6-3791k8s00000003-attachment)
I0922 05:06:11.324902    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:06:11.324902    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:06:11.324902    6408 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:06:11.402434    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:06:11.402434    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:06:13.710624    6408 azuredisk_v2.go:551]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-22T05:06:13Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="47e2bf47-3a34-11ed-be00-00155d071c25"
I0922 05:06:13.721471    6408 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/3791k8s00000003/status 200 OK in 10 milliseconds
I0922 05:06:13.721471    6408 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="47e2bf47-3a34-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10846200
I0922 05:06:31.152811    6408 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 05:06:31.152811    6408 utils.go:79] GRPC request: {}
I0922 05:06:31.152811    6408 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 05:06:36.154298    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:06:36.154866    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="45272d66-3a34-11ed-be00-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f76bc538-dc46-46e1-a42d-fc5c0ee737b6" "latency"=27027963500
I0922 05:06:36.154866    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="45272d66-3a34-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=27028531900
I0922 05:06:39.616004    6408 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-f76bc538-dc46-46e1-a42d-fc5c0ee737b6\globalmount with mount options([])
I0922 05:06:41.330334    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:06:41.330381    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:06:41.408283    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 186 lines ...
I0922 05:13:38.994894    6408 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 05:13:38.997125    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 05:13:38.997125    6408 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-d8d5515a-5ddc-4f87-826f-5c5517e7539b\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d8d5515a-5ddc-4f87-826f-5c5517e7539b","csi.storage.k8s.io/pvc/name":"pvc-rfxl9","csi.storage.k8s.io/pvc/namespace":"azuredisk-3867","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-d8d5515a-5ddc-4f87-826f-5c5517e7539b"}
I0922 05:13:38.997316    6408 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-d8d5515a-5ddc-4f87-826f-5c5517e7539b-3791k8s00000003-attachment)
I0922 05:13:41.409411    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:13:41.409411    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:13:41.409976    6408 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0922 05:13:41.612244    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:13:41.612244    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:13:43.723700    6408 azuredisk_v2.go:551]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-22T05:13:43Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="541d47b9-3a35-11ed-be00-00155d071c25"
I0922 05:13:43.734191    6408 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/3791k8s00000003/status 200 OK in 6 milliseconds
I0922 05:13:43.734191    6408 azuredisk_v2.go:538]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="541d47b9-3a35-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=10575800
I0922 05:13:45.594373    6408 reflector.go:536] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 42 items received
I0922 05:13:45.595435    6408 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=16102&timeout=6m46s&timeoutSeconds=406&watch=true 200 OK in 0 milliseconds
I0922 05:14:01.151320    6408 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0922 05:14:01.151320    6408 utils.go:79] GRPC request: {}
I0922 05:14:01.151320    6408 utils.go:85] GRPC response: {"ready":{"value":true}}
I0922 05:14:10.507901    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:14:10.508156    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="514c1aae-3a35-11ed-be00-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d8d5515a-5ddc-4f87-826f-5c5517e7539b" "latency"=31510189400
I0922 05:14:10.508232    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="514c1aae-3a35-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=31510385900
I0922 05:14:11.416552    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:14:11.416552    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:14:11.622089    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:14:11.622089    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 94 lines ...
I0922 05:17:04.192489    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0922 05:17:04.192489    6408 utils.go:79] GRPC request: {}
I0922 05:17:04.192489    6408 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 05:17:04.193807    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 05:17:04.193950    6408 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-fc274787-ab7a-4751-9d73-f934d576cb50\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-fc274787-ab7a-4751-9d73-f934d576cb50","csi.storage.k8s.io/pvc/name":"pvc-7hsct","csi.storage.k8s.io/pvc/namespace":"azuredisk-741","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-fc274787-ab7a-4751-9d73-f934d576cb50"}
I0922 05:17:04.194110    6408 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-fc274787-ab7a-4751-9d73-f934d576cb50-3791k8s00000003-attachment)
I0922 05:17:08.667090    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:17:08.667215    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="cb9aa674-3a35-11ed-be00-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-fc274787-ab7a-4751-9d73-f934d576cb50" "latency"=4472918100
I0922 05:17:08.667298    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="cb9aa674-3a35-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4473043100
I0922 05:17:11.448584    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:17:11.448584    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:17:11.648222    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:17:11.648222    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 66 lines ...
I0922 05:18:35.648482    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0922 05:18:35.648482    6408 utils.go:79] GRPC request: {}
I0922 05:18:35.648482    6408 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 05:18:35.652020    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 05:18:35.652020    6408 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-f426f608-0ffc-486a-afb4-0aca226d9da0\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-f426f608-0ffc-486a-afb4-0aca226d9da0","csi.storage.k8s.io/pvc/name":"pvc1-azuredisk-volume-tester-4w6fj-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-1959","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-f426f608-0ffc-486a-afb4-0aca226d9da0"}
I0922 05:18:35.652020    6408 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-f426f608-0ffc-486a-afb4-0aca226d9da0-3791k8s00000003-attachment)
I0922 05:18:38.557949    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:18:38.558055    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="021e050b-3a36-11ed-be00-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-f426f608-0ffc-486a-afb4-0aca226d9da0" "latency"=2905898800
I0922 05:18:38.558116    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="021e050b-3a36-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2906004400
I0922 05:18:41.485501    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:18:41.485501    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:18:41.668374    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:18:41.668374    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 223 lines ...
I0922 05:25:33.845072    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0922 05:25:33.845119    6408 utils.go:79] GRPC request: {}
I0922 05:25:33.845119    6408 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 05:25:33.846461    6408 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 05:25:33.846461    6408 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-495b02b3-0776-498a-b8a3-8c2434c639bf\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-495b02b3-0776-498a-b8a3-8c2434c639bf","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663820403388-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-hpczfw5p/providers/Microsoft.Compute/disks/pvc-495b02b3-0776-498a-b8a3-8c2434c639bf"}
I0922 05:25:33.846461    6408 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-495b02b3-0776-498a-b8a3-8c2434c639bf-3791k8s00000003-attachment)
I0922 05:25:38.048156    6408 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0922 05:25:38.048230    6408 conditionwaiter.go:50]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLunOrAttach" "disk.csi.azure.com/node-name"="3791k8s00000003" "disk.csi.azure.com/request-id"="fb616d67-3a36-11ed-be00-00155d071c25" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-495b02b3-0776-498a-b8a3-8c2434c639bf" "latency"=4201697100
I0922 05:25:38.048230    6408 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="fb616d67-3a36-11ed-be00-00155d071c25" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=4201697100
I0922 05:25:41.608011    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:25:41.608011    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0922 05:25:41.609345    6408 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-495b02b3-0776-498a-b8a3-8c2434c639bf\globalmount with mount options([])
I0922 05:25:41.803770    6408 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 95 lines ...
I0922 04:20:34.292431    6364 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1512&timeout=9m41s&timeoutSeconds=581&watch=true 200 OK in 1 milliseconds
I0922 04:20:34.292431    6364 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=1502&timeout=8m6s&timeoutSeconds=486&watch=true 200 OK in 1 milliseconds
I0922 04:20:34.393358    6364 shared_informer.go:285] caches populated
I0922 04:20:34.393358    6364 azuredisk_v2.go:193] driver userAgent: disk.csi.azure.com/latest-v2-1f682f0f8a1809a0520594ac2060ff2497311fad e2e-test
I0922 04:20:34.394177    6364 azure_disk_utils.go:543] reading cloud config from secret kube-system/azure-cloud-provider
I0922 04:20:34.400925    6364 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 5 milliseconds
I0922 04:20:34.405832    6364 azure_disk_utils.go:550] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0922 04:20:34.405832    6364 azure_disk_utils.go:555] could not read cloud config from secret kube-system/azure-cloud-provider
I0922 04:20:34.405832    6364 azure_disk_utils.go:565] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0922 04:20:34.413362    6364 azure_disk_utils.go:573] read cloud config from file: C:\k\azure.json successfully
I0922 04:20:34.415276    6364 azure_auth.go:245] Using AzurePublicCloud environment
I0922 04:20:34.415823    6364 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I0922 04:20:34.415823    6364 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
I0922 04:20:34.415823    6364 azure_auth.go:113] azure: User Assigned MSI ID is client ID. Resource ID parsing error: %+vparsing failed for d55c9a9f-4cdb-4884-bfff-b50f401b2560. Invalid resource Id format
I0922 04:20:34.423790    6364 azure_diskclient.go:68] Azure DisksClient using API version: 2021-04-01
I0922 04:20:34.426439    6364 azure.go:1011] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333333, Bucket: 80
I0922 04:20:34.431833    6364 safe_mounter_windows.go:406] using CSIProxyMounterV1, API Versions filesystem: v1, disk: v1, volume: v1
I0922 04:20:34.432060    6364 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0922 04:20:34.432060    6364 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0922 04:20:34.432060    6364 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
... skipping 100 lines ...
I0922 04:23:43.113812    6364 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0922 04:23:43.113812    6364 utils.go:79] GRPC request: {}
I0922 04:23:43.113812    6364 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0922 04:23:43.116271    6364 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0922 04:23:43.116271    6364 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-b74f159f-909e-45e2-9e9f-a2bedaa134e9\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"