This job view page is being replaced by Spyglass soon. Check out the new job view.
PRsunpa93: [V2] fix: CRI recovery should not overwrite existing values
ResultABORTED
Tests 0 failed / 25 succeeded
Started2022-09-20 15:05
Elapsed1h37m
Revision217d1f12b08c4f9cbd77ba2de8489be9df642bcd
Refs 1518

No Test Failures!


Show 25 Passed Tests

Show 82 Skipped Tests

Error lines from build-log.txt

... skipping 101 lines ...

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 11156  100 11156    0     0   162k      0 --:--:-- --:--:-- --:--:--  162k
Downloading https://get.helm.sh/helm-v3.9.4-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
docker pull k8sprow.azurecr.io/azuredisk-csi:latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 || make container-all push-manifest
Error response from daemon: manifest for k8sprow.azurecr.io/azuredisk-csi:latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 not found: manifest unknown: manifest tagged by "latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03" is not found
make[2]: Entering directory '/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver'
CGO_ENABLED=0 GOOS=windows go build -a -ldflags "-X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.driverVersion=latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.gitCommit=626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 -X sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.buildDate=2022-09-20T15:12:56Z -extldflags "-static"" -tags azurediskv2 -mod vendor -o _output/amd64/azurediskpluginv2.exe ./pkg/azurediskplugin
docker buildx rm container-builder || true
error: no builder "container-builder" found
docker buildx create --use --name=container-builder
container-builder
# enable qemu for arm64 build
# https://github.com/docker/buildx/issues/464#issuecomment-741507760
docker run --privileged --rm tonistiigi/binfmt --uninstall qemu-aarch64
Unable to find image 'tonistiigi/binfmt:latest' locally
... skipping 1790 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status Required
            properties:
              detail:
                description: Status summarizes the current attachment state of the
                  volume attachment Nil Status indicates that the volume has not yet
                  been attached to the node
                properties:
... skipping 7 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 90 lines ...
            - volumeName
            - volume_context
            - volume_id
            type: object
          status:
            description: status represents the current state of AzVolumeAttachment.
              includes error, state, and attachment status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 13 lines ...
                  role:
                    description: The current attachment role.
                    type: string
                required:
                - role
                type: object
              error:
                description: Error occurred during attach/detach of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 169 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              detail:
                description: Current status detail of the AzVolume Nil detail indicates
                  that the volume has not been created
                properties:
                  accessible_topology:
... skipping 28 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 154 lines ...
            - maxMountReplicaCount
            - volumeCapability
            - volumeName
            type: object
          status:
            description: status represents the current state of AzVolume. includes
              error, state, and volume status
            properties:
              annotation:
                additionalProperties:
                  type: string
                description: Annotations contains additional resource information
                  to guide driver actions
... skipping 34 lines ...
                    type: string
                required:
                - capacity_bytes
                - node_expansion_required
                - volume_id
                type: object
              error:
                description: Error occurred during creation/deletion of volume
                properties:
                  code:
                    type: string
                  message:
                    type: string
                  parameters:
... skipping 773 lines ...
          image: "mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0"
          args:
            - "-csi-address=$(ADDRESS)"
            - "-v=2"
            - "-leader-election"
            - "--leader-election-namespace=kube-system"
            - '-handle-volume-inuse-error=false'
            - '-feature-gates=RecoverVolumeExpansionFailure=true'
            - "-timeout=240s"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
          volumeMounts:
... skipping 226 lines ...
Sep 20 15:19:56.091: INFO: Found ClusterRoles; assuming RBAC is enabled.
STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-1410
STEP: Waiting for a default service account to be provisioned in namespace
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace
I0920 15:19:56.591919   15385 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 20 15:19:56.732: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-vtbkh" in namespace "azuredisk-1410" to be "Succeeded or Failed"
Sep 20 15:19:56.789: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Pending", Reason="", readiness=false. Elapsed: 57.237679ms
Sep 20 15:19:58.848: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.116523888s
Sep 20 15:20:00.905: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.17347575s
Sep 20 15:20:02.964: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.232112785s
Sep 20 15:20:05.022: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.290456573s
Sep 20 15:20:07.079: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Pending", Reason="", readiness=false. Elapsed: 10.347440417s
... skipping 4 lines ...
Sep 20 15:20:17.372: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Running", Reason="", readiness=true. Elapsed: 20.640317509s
Sep 20 15:20:19.430: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Running", Reason="", readiness=true. Elapsed: 22.698168916s
Sep 20 15:20:21.491: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Running", Reason="", readiness=true. Elapsed: 24.759076614s
Sep 20 15:20:23.549: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Running", Reason="", readiness=false. Elapsed: 26.817528358s
Sep 20 15:20:25.607: INFO: Pod "azuredisk-volume-tester-vtbkh": Phase="Succeeded", Reason="", readiness=false. Elapsed: 28.875378184s
STEP: Saw pod success
Sep 20 15:20:25.608: INFO: Pod "azuredisk-volume-tester-vtbkh" satisfied condition "Succeeded or Failed"
Sep 20 15:20:25.608: INFO: deleting Pod "azuredisk-1410"/"azuredisk-volume-tester-vtbkh"
Sep 20 15:20:25.704: INFO: Pod azuredisk-volume-tester-vtbkh has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-vtbkh in namespace azuredisk-1410
Sep 20 15:20:25.769: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "azuredisk-1410" for this suite.
... skipping 18 lines ...
I0920 15:20:26.563318   15385 azuredisk_driver.go:52] Using azure disk driver: disk.csi.azure.com
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 20 15:20:26.805: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-r4l8n" in namespace "azuredisk-1884" to be "Succeeded or Failed"
Sep 20 15:20:26.874: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Pending", Reason="", readiness=false. Elapsed: 68.324636ms
Sep 20 15:20:28.932: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Pending", Reason="", readiness=false. Elapsed: 2.126843388s
Sep 20 15:20:30.992: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Pending", Reason="", readiness=false. Elapsed: 4.18650831s
Sep 20 15:20:33.050: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Pending", Reason="", readiness=false. Elapsed: 6.244919879s
Sep 20 15:20:35.108: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Pending", Reason="", readiness=false. Elapsed: 8.303097424s
Sep 20 15:20:37.167: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Pending", Reason="", readiness=false. Elapsed: 10.361999225s
... skipping 27 lines ...
Sep 20 15:21:34.809: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Running", Reason="", readiness=true. Elapsed: 1m8.00347962s
Sep 20 15:21:36.867: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Running", Reason="", readiness=true. Elapsed: 1m10.06190702s
Sep 20 15:21:38.926: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Running", Reason="", readiness=true. Elapsed: 1m12.120549397s
Sep 20 15:21:40.985: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Running", Reason="", readiness=false. Elapsed: 1m14.179271461s
Sep 20 15:21:43.042: INFO: Pod "azuredisk-volume-tester-r4l8n": Phase="Succeeded", Reason="", readiness=false. Elapsed: 1m16.236930095s
STEP: Saw pod success
Sep 20 15:21:43.042: INFO: Pod "azuredisk-volume-tester-r4l8n" satisfied condition "Succeeded or Failed"
Sep 20 15:21:43.042: INFO: deleting Pod "azuredisk-1884"/"azuredisk-volume-tester-r4l8n"
Sep 20 15:21:43.144: INFO: Pod azuredisk-volume-tester-r4l8n has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-r4l8n in namespace azuredisk-1884
STEP: validating provisioned PV
STEP: checking the PV
... skipping 124 lines ...
STEP: creating a PVC
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 20 15:29:14.205: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gctbg" in namespace "azuredisk-9164" to be "Succeeded or Failed"
Sep 20 15:29:14.281: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Pending", Reason="", readiness=false. Elapsed: 75.992321ms
Sep 20 15:29:16.339: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Pending", Reason="", readiness=false. Elapsed: 2.133680614s
Sep 20 15:29:18.396: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Pending", Reason="", readiness=false. Elapsed: 4.191234001s
Sep 20 15:29:20.455: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Pending", Reason="", readiness=false. Elapsed: 6.249846674s
Sep 20 15:29:22.512: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Pending", Reason="", readiness=false. Elapsed: 8.306786283s
Sep 20 15:29:24.570: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Pending", Reason="", readiness=false. Elapsed: 10.364725011s
... skipping 18 lines ...
Sep 20 15:30:03.681: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Running", Reason="", readiness=true. Elapsed: 49.475830486s
Sep 20 15:30:05.739: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Running", Reason="", readiness=true. Elapsed: 51.534441679s
Sep 20 15:30:07.798: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Running", Reason="", readiness=false. Elapsed: 53.593121717s
Sep 20 15:30:09.855: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Running", Reason="", readiness=false. Elapsed: 55.650115924s
Sep 20 15:30:11.913: INFO: Pod "azuredisk-volume-tester-gctbg": Phase="Succeeded", Reason="", readiness=false. Elapsed: 57.707827711s
STEP: Saw pod success
Sep 20 15:30:11.913: INFO: Pod "azuredisk-volume-tester-gctbg" satisfied condition "Succeeded or Failed"
Sep 20 15:30:11.913: INFO: deleting Pod "azuredisk-9164"/"azuredisk-volume-tester-gctbg"
Sep 20 15:30:12.012: INFO: Pod azuredisk-volume-tester-gctbg has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-gctbg in namespace azuredisk-9164
STEP: validating provisioned PV
STEP: checking the PV
... skipping 431 lines ...
STEP: checking the PVC
STEP: validating provisioned PV
STEP: checking the PV
STEP: setting up the pod
I0920 15:43:58.153065   15385 resource_setup.go:63] adding PV (pvc-95fd211e-2f45-48b0-a867-934b746e5775) to pod ()
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 20 15:43:58.246: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-vwhdv" in namespace "azuredisk-9353" to be "Succeeded or Failed"
Sep 20 15:43:58.330: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Pending", Reason="", readiness=false. Elapsed: 83.750533ms
Sep 20 15:44:00.415: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.168919968s
Sep 20 15:44:02.505: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.258336107s
Sep 20 15:44:04.593: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.34667112s
Sep 20 15:44:06.675: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.428595046s
Sep 20 15:44:08.766: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.520165304s
... skipping 15 lines ...
Sep 20 15:44:42.202: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Running", Reason="", readiness=true. Elapsed: 43.955743786s
Sep 20 15:44:44.288: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Running", Reason="", readiness=true. Elapsed: 46.041523348s
Sep 20 15:44:46.379: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Running", Reason="", readiness=false. Elapsed: 48.132245097s
Sep 20 15:44:48.467: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Running", Reason="", readiness=false. Elapsed: 50.220493297s
Sep 20 15:44:50.555: INFO: Pod "azuredisk-volume-tester-vwhdv": Phase="Succeeded", Reason="", readiness=false. Elapsed: 52.308784373s
STEP: Saw pod success
Sep 20 15:44:50.555: INFO: Pod "azuredisk-volume-tester-vwhdv" satisfied condition "Succeeded or Failed"
Sep 20 15:44:50.555: INFO: deleting Pod "azuredisk-9353"/"azuredisk-volume-tester-vwhdv"
Sep 20 15:44:50.699: INFO: Pod azuredisk-volume-tester-vwhdv has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-vwhdv in namespace azuredisk-9353
Sep 20 15:44:50.794: INFO: deleting PVC "azuredisk-9353"/"pvc-mdwt2"
Sep 20 15:44:50.794: INFO: Deleting PersistentVolumeClaim "pvc-mdwt2"
... skipping 46 lines ...
Sep 20 15:45:36.053: INFO: PersistentVolumeClaim pvc-q6twn found but phase is Pending instead of Bound.
Sep 20 15:45:38.143: INFO: PersistentVolumeClaim pvc-q6twn found and phase=Bound (4.262166401s)
STEP: checking the PVC
STEP: validating provisioned PV
STEP: checking the PV
STEP: deploying the pod
STEP: checking that the pods command exits with no error
Sep 20 15:45:38.416: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-htddz" in namespace "azuredisk-6754" to be "Succeeded or Failed"
Sep 20 15:45:38.505: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Pending", Reason="", readiness=false. Elapsed: 88.5805ms
Sep 20 15:45:40.590: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Pending", Reason="", readiness=false. Elapsed: 2.173711336s
Sep 20 15:45:42.684: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Pending", Reason="", readiness=false. Elapsed: 4.267148934s
Sep 20 15:45:44.774: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Pending", Reason="", readiness=false. Elapsed: 6.357443556s
Sep 20 15:45:46.862: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Pending", Reason="", readiness=false. Elapsed: 8.445402067s
Sep 20 15:45:48.948: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Pending", Reason="", readiness=false. Elapsed: 10.531340337s
... skipping 14 lines ...
Sep 20 15:46:20.394: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Pending", Reason="", readiness=false. Elapsed: 41.977187716s
Sep 20 15:46:22.484: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Running", Reason="", readiness=true. Elapsed: 44.067617857s
Sep 20 15:46:24.572: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Running", Reason="", readiness=false. Elapsed: 46.155762466s
Sep 20 15:46:26.666: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Running", Reason="", readiness=false. Elapsed: 48.249610229s
Sep 20 15:46:28.754: INFO: Pod "azuredisk-volume-tester-htddz": Phase="Succeeded", Reason="", readiness=false. Elapsed: 50.338001697s
STEP: Saw pod success
Sep 20 15:46:28.755: INFO: Pod "azuredisk-volume-tester-htddz" satisfied condition "Succeeded or Failed"
Sep 20 15:46:28.755: INFO: deleting Pod "azuredisk-6754"/"azuredisk-volume-tester-htddz"
Sep 20 15:46:28.885: INFO: Pod azuredisk-volume-tester-htddz has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-htddz in namespace azuredisk-6754
Sep 20 15:46:28.983: INFO: deleting PVC "azuredisk-6754"/"pvc-q6twn"
Sep 20 15:46:28.983: INFO: Deleting PersistentVolumeClaim "pvc-q6twn"
... skipping 39 lines ...
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: setting up the pod
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 20 15:47:12.171: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-9sb9h" in namespace "azuredisk-4070" to be "Succeeded or Failed"
Sep 20 15:47:12.255: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Pending", Reason="", readiness=false. Elapsed: 83.943971ms
Sep 20 15:47:14.346: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Pending", Reason="", readiness=false. Elapsed: 2.174387886s
Sep 20 15:47:16.434: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Pending", Reason="", readiness=false. Elapsed: 4.263037954s
Sep 20 15:47:18.520: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Pending", Reason="", readiness=false. Elapsed: 6.348869765s
Sep 20 15:47:20.618: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Pending", Reason="", readiness=false. Elapsed: 8.446462871s
Sep 20 15:47:22.705: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Pending", Reason="", readiness=false. Elapsed: 10.533231129s
... skipping 16 lines ...
Sep 20 15:47:58.220: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Pending", Reason="", readiness=false. Elapsed: 46.048525155s
Sep 20 15:48:00.307: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Running", Reason="", readiness=true. Elapsed: 48.135312915s
Sep 20 15:48:02.399: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Running", Reason="", readiness=false. Elapsed: 50.227905933s
Sep 20 15:48:04.487: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Running", Reason="", readiness=false. Elapsed: 52.315513354s
Sep 20 15:48:06.577: INFO: Pod "azuredisk-volume-tester-9sb9h": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.40566232s
STEP: Saw pod success
Sep 20 15:48:06.577: INFO: Pod "azuredisk-volume-tester-9sb9h" satisfied condition "Succeeded or Failed"
Sep 20 15:48:06.577: INFO: deleting Pod "azuredisk-4070"/"azuredisk-volume-tester-9sb9h"
Sep 20 15:48:06.726: INFO: Pod azuredisk-volume-tester-9sb9h has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-9sb9h in namespace azuredisk-4070
STEP: validating provisioned PV
STEP: checking the PV
... skipping 41 lines ...
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: setting up the pod
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 20 15:48:50.260: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-d56rp" in namespace "azuredisk-1488" to be "Succeeded or Failed"
Sep 20 15:48:50.348: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Pending", Reason="", readiness=false. Elapsed: 88.730105ms
Sep 20 15:48:52.442: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Pending", Reason="", readiness=false. Elapsed: 2.182550112s
Sep 20 15:48:54.529: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Pending", Reason="", readiness=false. Elapsed: 4.269646504s
Sep 20 15:48:56.625: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Pending", Reason="", readiness=false. Elapsed: 6.364849962s
Sep 20 15:48:58.719: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Pending", Reason="", readiness=false. Elapsed: 8.45901376s
Sep 20 15:49:00.807: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Pending", Reason="", readiness=false. Elapsed: 10.54720394s
... skipping 16 lines ...
Sep 20 15:49:36.301: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Pending", Reason="", readiness=false. Elapsed: 46.041641966s
Sep 20 15:49:38.386: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Running", Reason="", readiness=true. Elapsed: 48.126592212s
Sep 20 15:49:40.481: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Running", Reason="", readiness=false. Elapsed: 50.221326412s
Sep 20 15:49:42.571: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Running", Reason="", readiness=false. Elapsed: 52.311777479s
Sep 20 15:49:44.660: INFO: Pod "azuredisk-volume-tester-d56rp": Phase="Succeeded", Reason="", readiness=false. Elapsed: 54.400716768s
STEP: Saw pod success
Sep 20 15:49:44.660: INFO: Pod "azuredisk-volume-tester-d56rp" satisfied condition "Succeeded or Failed"
Sep 20 15:49:44.660: INFO: deleting Pod "azuredisk-1488"/"azuredisk-volume-tester-d56rp"
Sep 20 15:49:44.788: INFO: Pod azuredisk-volume-tester-d56rp has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-d56rp in namespace azuredisk-1488
STEP: validating provisioned PV
STEP: checking the PV
... skipping 108 lines ...
STEP: checking the PVC
STEP: validating provisioned PV
STEP: checking the PV
STEP: setting up the pod
I0920 15:50:36.343965   15385 resource_setup.go:63] adding PV (pvc-7df4bb97-9aee-43b5-87dd-2c823ff7d85e) to pod ()
STEP: deploying the pod
STEP: checking that the pod's command exits with an error
Sep 20 15:50:36.435: INFO: Waiting up to 10m0s for pod "azuredisk-volume-tester-9nwts" in namespace "azuredisk-3103" to be "Error status code"
Sep 20 15:50:36.521: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Pending", Reason="", readiness=false. Elapsed: 85.602346ms
Sep 20 15:50:38.616: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Pending", Reason="", readiness=false. Elapsed: 2.181406954s
Sep 20 15:50:40.705: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Pending", Reason="", readiness=false. Elapsed: 4.270256444s
Sep 20 15:50:42.793: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Pending", Reason="", readiness=false. Elapsed: 6.357894051s
Sep 20 15:50:44.879: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Pending", Reason="", readiness=false. Elapsed: 8.443694267s
Sep 20 15:50:46.964: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Pending", Reason="", readiness=false. Elapsed: 10.528980125s
... skipping 25 lines ...
Sep 20 15:51:41.274: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Pending", Reason="", readiness=false. Elapsed: 1m4.838972122s
Sep 20 15:51:43.365: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Pending", Reason="", readiness=false. Elapsed: 1m6.929564808s
Sep 20 15:51:45.456: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Running", Reason="", readiness=true. Elapsed: 1m9.021042157s
Sep 20 15:51:47.545: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Running", Reason="", readiness=true. Elapsed: 1m11.109785575s
Sep 20 15:51:49.634: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Running", Reason="", readiness=false. Elapsed: 1m13.198636056s
Sep 20 15:51:51.725: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Running", Reason="", readiness=false. Elapsed: 1m15.289707272s
Sep 20 15:51:53.806: INFO: Pod "azuredisk-volume-tester-9nwts": Phase="Failed", Reason="", readiness=false. Elapsed: 1m17.370686693s
STEP: Saw pod failure
Sep 20 15:51:53.806: INFO: Pod "azuredisk-volume-tester-9nwts" satisfied condition "Error status code"
STEP: checking that pod logs contain expected message
Sep 20 15:51:53.934: INFO: deleting Pod "azuredisk-3103"/"azuredisk-volume-tester-9nwts"
Sep 20 15:51:54.025: INFO: Pod azuredisk-volume-tester-9nwts has the following logs: out-file : Access to the path 'C:\mnt\test-1\data' is denied.
At line:1 char:1
+ echo $null >> C:\mnt\test-1\data
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
... skipping 455 lines ...
STEP: checking the PVC
STEP: validating provisioned PV
STEP: checking the PV
STEP: setting up the pod
I0920 15:59:27.406733   15385 resource_setup.go:63] adding PV (pvc-fe36a319-4b65-4e5c-9b6a-1eb11b36c3ee) to pod ()
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 20 15:59:27.502: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-gl727" in namespace "azuredisk-852" to be "Succeeded or Failed"
Sep 20 15:59:27.607: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Pending", Reason="", readiness=false. Elapsed: 105.342176ms
Sep 20 15:59:29.695: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Pending", Reason="", readiness=false. Elapsed: 2.19297719s
Sep 20 15:59:31.792: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Pending", Reason="", readiness=false. Elapsed: 4.290265482s
Sep 20 15:59:33.886: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Pending", Reason="", readiness=false. Elapsed: 6.384057942s
Sep 20 15:59:35.977: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Pending", Reason="", readiness=false. Elapsed: 8.474413583s
Sep 20 15:59:38.069: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Pending", Reason="", readiness=false. Elapsed: 10.56645138s
... skipping 14 lines ...
Sep 20 16:00:09.428: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Pending", Reason="", readiness=false. Elapsed: 41.925647796s
Sep 20 16:00:11.516: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Pending", Reason="", readiness=false. Elapsed: 44.013426502s
Sep 20 16:00:13.612: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Running", Reason="", readiness=true. Elapsed: 46.109470843s
Sep 20 16:00:15.699: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Running", Reason="", readiness=false. Elapsed: 48.197108491s
Sep 20 16:00:17.789: INFO: Pod "azuredisk-volume-tester-gl727": Phase="Succeeded", Reason="", readiness=false. Elapsed: 50.287044176s
STEP: Saw pod success
Sep 20 16:00:17.789: INFO: Pod "azuredisk-volume-tester-gl727" satisfied condition "Succeeded or Failed"
Sep 20 16:00:17.789: INFO: deleting Pod "azuredisk-852"/"azuredisk-volume-tester-gl727"
Sep 20 16:00:17.918: INFO: Pod azuredisk-volume-tester-gl727 has the following logs: hello world
hello world
hello world

STEP: Deleting pod azuredisk-volume-tester-gl727 in namespace azuredisk-852
... skipping 155 lines ...
STEP: creating a PVC
STEP: setting up the StorageClass
STEP: creating a StorageClass 
STEP: setting up the PVC and PV
STEP: creating a PVC
STEP: deploying the pod
STEP: checking that the pod's command exits with no error
Sep 20 16:01:28.226: INFO: Waiting up to 15m0s for pod "azuredisk-volume-tester-tkdcr" in namespace "azuredisk-7039" to be "Succeeded or Failed"
Sep 20 16:01:28.320: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Pending", Reason="", readiness=false. Elapsed: 94.206205ms
Sep 20 16:01:30.411: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Pending", Reason="", readiness=false. Elapsed: 2.184750727s
Sep 20 16:01:32.520: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Pending", Reason="", readiness=false. Elapsed: 4.293909713s
Sep 20 16:01:34.610: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Pending", Reason="", readiness=false. Elapsed: 6.383474454s
Sep 20 16:01:36.700: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Pending", Reason="", readiness=false. Elapsed: 8.474126445s
Sep 20 16:01:38.783: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Pending", Reason="", readiness=false. Elapsed: 10.556643227s
... skipping 18 lines ...
Sep 20 16:02:18.483: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Running", Reason="", readiness=true. Elapsed: 50.256965985s
Sep 20 16:02:20.575: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Running", Reason="", readiness=false. Elapsed: 52.348540271s
Sep 20 16:02:22.657: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Running", Reason="", readiness=false. Elapsed: 54.431037871s
Sep 20 16:02:24.748: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Running", Reason="", readiness=false. Elapsed: 56.521261599s
Sep 20 16:02:26.838: INFO: Pod "azuredisk-volume-tester-tkdcr": Phase="Succeeded", Reason="", readiness=false. Elapsed: 58.611436045s
STEP: Saw pod success
Sep 20 16:02:26.838: INFO: Pod "azuredisk-volume-tester-tkdcr" satisfied condition "Succeeded or Failed"
Sep 20 16:02:26.838: INFO: deleting Pod "azuredisk-7039"/"azuredisk-volume-tester-tkdcr"
Sep 20 16:02:26.971: INFO: Pod azuredisk-volume-tester-tkdcr has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-tkdcr in namespace azuredisk-7039
STEP: validating provisioned PV
STEP: checking the PV
... skipping 349 lines ...
STEP: begin to delete the pod
Sep 20 16:12:25.880: INFO: deleting Pod "azuredisk-575"/"azuredisk-volume-tester-lbzml"
Sep 20 16:12:25.974: INFO: Pod azuredisk-volume-tester-lbzml has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-lbzml in namespace azuredisk-575
Sep 20 16:12:58.155: INFO: deleting Pod "azuredisk-575"/"azuredisk-volume-tester-lbzml"
Sep 20 16:12:58.215: INFO: Error getting logs for pod azuredisk-volume-tester-lbzml: the server could not find the requested resource (get pods azuredisk-volume-tester-lbzml)
STEP: Deleting pod azuredisk-volume-tester-lbzml in namespace azuredisk-575
STEP: validating provisioned PV
STEP: checking the PV
Sep 20 16:12:58.392: INFO: deleting PVC "azuredisk-575"/"pvc-t5tmp"
Sep 20 16:12:58.392: INFO: Deleting PersistentVolumeClaim "pvc-t5tmp"
STEP: waiting for claim's PV "pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" to be deleted
... skipping 43 lines ...
STEP: begin to delete the pod
Sep 20 16:13:55.553: INFO: deleting Pod "azuredisk-6003"/"azuredisk-volume-tester-vq9xl"
Sep 20 16:13:55.665: INFO: Pod azuredisk-volume-tester-vq9xl has the following logs: hello world

STEP: Deleting pod azuredisk-volume-tester-vq9xl in namespace azuredisk-6003
Sep 20 16:14:13.853: INFO: deleting Pod "azuredisk-6003"/"azuredisk-volume-tester-vq9xl"
Sep 20 16:14:13.909: INFO: Error getting logs for pod azuredisk-volume-tester-vq9xl: the server could not find the requested resource (get pods azuredisk-volume-tester-vq9xl)
STEP: Deleting pod azuredisk-volume-tester-vq9xl in namespace azuredisk-6003
STEP: validating provisioned PV
STEP: checking the PV
Sep 20 16:14:14.082: INFO: deleting PVC "azuredisk-6003"/"pvc-cvlrf"
Sep 20 16:14:14.082: INFO: Deleting PersistentVolumeClaim "pvc-cvlrf"
STEP: waiting for claim's PV "pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" to be deleted
... skipping 514 lines ...

    test case not supported by Windows clusters

    /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/utils/testutil/env_utils.go:26
------------------------------
Dynamic Provisioning [multi-az] 
  should check failed replica attachments are recreated after space is made from a volume detaching.
  /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1764
STEP: Creating a kubernetes client
Sep 20 16:20:58.451: INFO: >>> kubeConfig: /root/tmp3045723636/kubeconfig/kubeconfig.westus2.json
STEP: Building a namespace api object, basename azuredisk
STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in azuredisk-5472
STEP: Waiting for a default service account to be provisioned in namespace
... skipping 8 lines ...

S [SKIPPING] [1.268 seconds]
Dynamic Provisioning
/home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:40
  [multi-az]
  /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:48
    should check failed replica attachments are recreated after space is made from a volume detaching. [It]
    /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1764

    Skip flakey test until #1378 is fixed.

    /home/prow/go/src/sigs.k8s.io/azuredisk-csi-driver/test/e2e/dynamic_provisioning_test.go:1773
------------------------------
... skipping 102 lines ...
I0920 15:19:01.826694       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=1621&timeout=5m1s&timeoutSeconds=301&watch=true 200 OK in 2 milliseconds
I0920 15:19:01.828761       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1620&timeout=6m12s&timeoutSeconds=372&watch=true 200 OK in 3 milliseconds
I0920 15:19:01.905936       1 shared_informer.go:285] caches populated
I0920 15:19:01.905983       1 azuredisk_v2.go:191] driver userAgent: disk.csi.azure.com/latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 e2e-test
I0920 15:19:01.905998       1 azure_disk_utils.go:543] reading cloud config from secret kube-system/azure-cloud-provider
I0920 15:19:01.908932       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 2 milliseconds
I0920 15:19:01.909246       1 azure_disk_utils.go:550] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0920 15:19:01.909269       1 azure_disk_utils.go:555] could not read cloud config from secret kube-system/azure-cloud-provider
I0920 15:19:01.909279       1 azure_disk_utils.go:565] use default AZURE_CREDENTIAL_FILE env var: /etc/kubernetes/azure.json
I0920 15:19:01.909312       1 azure_disk_utils.go:573] read cloud config from file: /etc/kubernetes/azure.json successfully
I0920 15:19:01.910237       1 azure_auth.go:245] Using AzurePublicCloud environment
I0920 15:19:01.910267       1 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I0920 15:19:01.910276       1 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
I0920 15:19:01.910330       1 azure_auth.go:113] azure: User Assigned MSI ID is client ID. Resource ID parsing error: %+vparsing failed for 0f481d95-6b3d-4086-b24e-9836129cae7a. Invalid resource Id format
I0920 15:19:01.910378       1 azure.go:781] Azure cloudprovider using try backoff: retries=6, exponent=1.500000, duration=5, jitter=1.000000
I0920 15:19:01.910447       1 azure_interfaceclient.go:72] Azure InterfacesClient (read ops) using rate limit config: QPS=100, bucket=1000
I0920 15:19:01.910460       1 azure_interfaceclient.go:75] Azure InterfacesClient (write ops) using rate limit config: QPS=100, bucket=1000
I0920 15:19:01.910492       1 azure_vmsizeclient.go:68] Azure VirtualMachineSizesClient (read ops) using rate limit config: QPS=100, bucket=1000
I0920 15:19:01.910514       1 azure_vmsizeclient.go:71] Azure VirtualMachineSizesClient (write ops) using rate limit config: QPS=100, bucket=1000
I0920 15:19:01.910541       1 azure_storageaccountclient.go:70] Azure StorageAccountClient (read ops) using rate limit config: QPS=100, bucket=1000
... skipping 26 lines ...
I0920 15:19:01.911256       1 skus.go:123] NewNodeInfo: Starting to populate node and disk sku information.
I0920 15:19:01.987112       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.075785918 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:19:01.987166       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:19:01.987189       1 azure_vmss_cache.go:332] Node  has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:19:02.004371       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017140249 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:19:02.004911       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
E0920 15:19:02.004949       1 azuredisk_v2.go:238] Failed to get node info. Error: NewNodeInfo: Failed to get instance type from Azure cloud provider, nodeName: , error: not a vmss instance
I0920 15:19:02.005333       1 mount_linux.go:208] Detected OS without systemd
I0920 15:19:02.005426       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0920 15:19:02.005536       1 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0920 15:19:02.005607       1 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0920 15:19:02.005714       1 driver.go:81] Enabling controller service capability: CLONE_VOLUME
I0920 15:19:02.005781       1 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
... skipping 342 lines ...
I0920 15:19:02.546322       1 shared_state.go:483]  "msg"="Storing pod kube-addon-manager-k8s-master-14178977-vmss000000 and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766"
I0920 15:19:02.546357       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/kube-addon-manager-k8s-master-14178977-vmss000000." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/kube-addon-manager-k8s-master-14178977-vmss000000"
I0920 15:19:02.546380       1 shared_state.go:307]  "msg"="Getting requested volumes for pod (kube-system/kube-addon-manager-k8s-master-14178977-vmss000000)." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/kube-addon-manager-k8s-master-14178977-vmss000000"
I0920 15:19:02.546417       1 pod.go:99]  "msg"="Pod kube-system/kube-addon-manager-k8s-master-14178977-vmss000000 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/kube-addon-manager-k8s-master-14178977-vmss000000"
I0920 15:19:02.546461       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/kube-addon-manager-k8s-master-14178977-vmss000000" "latency"=98601
I0920 15:19:02.546508       1 shared_state.go:413]  "msg"="Adding pod csi-azuredisk-controller-6f9f9c69fc-4t2ss to shared map with keyName kube-system/csi-azuredisk-controller-6f9f9c69fc-4t2ss." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766"
I0920 15:19:02.547081       1 shared_state.go:419]  "msg"="Pod spec of pod csi-azuredisk-controller-6f9f9c69fc-4t2ss is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-h4x8f VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=15s --leader-election --leader-election-namespace=kube-system --worker-threads=40 --extra-create-metadata=true --strict-topology=true] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc00057e7f0 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-master-14178977-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00057e7f8} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc00057e800}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc00057e808 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc00057e80c PreemptionPolicy:0xc0009f7600 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-h4x8f VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766"
I0920 15:19:02.547162       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6f9f9c69fc-4t2ss: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766"
I0920 15:19:02.547209       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6f9f9c69fc-4t2ss: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766"
I0920 15:19:02.547279       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6f9f9c69fc-4t2ss: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766"
I0920 15:19:02.547360       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6f9f9c69fc-4t2ss: Skipping Volume {kube-api-access-h4x8f {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766"
I0920 15:19:02.547387       1 shared_state.go:483]  "msg"="Storing pod csi-azuredisk-controller-6f9f9c69fc-4t2ss and claim [] to podToClaimsMap map." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766"
I0920 15:19:02.547436       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-azuredisk-controller-6f9f9c69fc-4t2ss." "disk.csi.azure.com/request-id"="8ef3734d-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-controller-6f9f9c69fc-4t2ss"
... skipping 134 lines ...
I0920 15:19:02.668957       1 shared_state.go:457]  "msg"="Pod kube-apiserver-k8s-master-14178977-vmss000000: Skipping Volume {auditlog {&HostPathVolumeSource{Path:/var/log/kubeaudit,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0920 15:19:02.668985       1 shared_state.go:483]  "msg"="Storing pod kube-apiserver-k8s-master-14178977-vmss000000 and claim [] to podToClaimsMap map." 
I0920 15:19:02.669040       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/kube-apiserver-k8s-master-14178977-vmss000000." "disk.csi.azure.com/request-id"="8f110c07-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/kube-apiserver-k8s-master-14178977-vmss000000"
I0920 15:19:02.669077       1 shared_state.go:307]  "msg"="Getting requested volumes for pod (kube-system/kube-apiserver-k8s-master-14178977-vmss000000)." "disk.csi.azure.com/request-id"="8f110c07-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/kube-apiserver-k8s-master-14178977-vmss000000"
I0920 15:19:02.669112       1 pod.go:99]  "msg"="Pod kube-system/kube-apiserver-k8s-master-14178977-vmss000000 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="8f110c07-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/kube-apiserver-k8s-master-14178977-vmss000000"
I0920 15:19:02.669153       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="8f110c07-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/kube-apiserver-k8s-master-14178977-vmss000000" "latency"=124101
I0920 15:19:02.670529       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="k8s-master-14178977-vmss000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0920 15:19:02.670587       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="8f114869-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=10600
I0920 15:19:02.670693       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="1417k8s00000003" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0920 15:19:02.670751       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="8f114f06-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=20500
I0920 15:19:02.670859       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="1417k8s00000001" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0920 15:19:02.670913       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="8f115510-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8800
I0920 15:19:02.671012       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="1417k8s00000002" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0920 15:19:02.671076       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="8f115b86-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=9200
I0920 15:19:02.671176       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="1417k8s00000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0920 15:19:02.671229       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="8f116189-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=8800
I0920 15:19:02.652649       1 shared_state.go:413]  "msg"="Adding pod csi-azuredisk-node-win-sl85v to shared map with keyName kube-system/csi-azuredisk-node-win-sl85v." 
I0920 15:19:02.652773       1 shared_state.go:413]  "msg"="Adding pod csi-azuredisk-node-win-c49lh to shared map with keyName kube-system/csi-azuredisk-node-win-c49lh." 
I0920 15:19:02.674397       1 shared_state.go:419]  "msg"="Pod spec of pod csi-azuredisk-node-win-c49lh is: {Volumes:[{Name:csi-proxy-fs-pipe-v1 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-filesystem-v1,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-disk-pipe-v1 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-disk-v1,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-volume-pipe-v1 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-volume-v1,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-node-win-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-fs-pipe-v1beta1 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-filesystem-v1beta1,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-disk-pipe-v1beta2 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-disk-v1beta2,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-volume-pipe-v1beta2 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-volume-v1beta2,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:registration-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:C:\\var\\lib\\kubelet\\plugins_registry\\,Type:*Directory,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kubelet-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:C:\\var\\lib\\kubelet\\,Type:*Directory,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:plugin-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:C:\\var\\lib\\kubelet\\plugins\\disk.csi.azure.com\\,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-config VolumeSource:{HostPath:&HostPathVolumeSource{Path:C:\\k,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-dzv4j VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=$(CSI_ENDPOINT) --probe-timeout=3s --health-port=29603 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:CSI_ENDPOINT Value:unix://C:\\\\csi\\\\csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:41943040 scale:0} d:{Dec:<nil>} s: Format:BinarySI}]} VolumeMounts:[{Name:plugin-dir ReadOnly:false MountPath:C:\\csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-dzv4j ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:node-driver-registrar Image:mcr.microsoft.com/oss/kubernetes-csi/csi-node-driver-registrar:v2.5.1 Command:[] Args:[--v=2 --csi-address=$(CSI_ENDPOINT) --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:CSI_ENDPOINT Value:unix://C:\\\\csi\\\\csi.sock ValueFrom:nil} {Name:DRIVER_REG_SOCK_PATH Value:C:\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\disk.csi.azure.com\\\\csi.sock ValueFrom:nil} {Name:KUBE_NODE_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:30 scale:-3} d:{Dec:<nil>} s:30m Format:DecimalSI} memory:{i:{value:41943040 scale:0} d:{Dec:<nil>} s: Format:BinarySI}]} VolumeMounts:[{Name:kubelet-dir ReadOnly:false MountPath:C:\\var\\lib\\kubelet SubPath: MountPropagation:<nil> SubPathExpr:} {Name:plugin-dir ReadOnly:false MountPath:C:\\csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:registration-dir ReadOnly:false MountPath:C:\\registration SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-dzv4j ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[/csi-node-driver-registrar.exe --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) --mode=kubelet-registration-probe],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:30,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 Command:[] Args:[--v=6 --nodeid=$(KUBE_NODE_NAME) --config=.\\etc\\csi-azuredisk-node-win\\config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:0 ContainerPort:29603 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path-windows,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix://C:\\\\csi\\\\csi.sock ValueFrom:nil} {Name:KUBE_NODE_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:spec.nodeName,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:209715200 scale:0} d:{Dec:<nil>} s: Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:41943040 scale:0} d:{Dec:<nil>} s: Format:BinarySI}]} VolumeMounts:[{Name:kubelet-dir ReadOnly:false MountPath:C:\\var\\lib\\kubelet SubPath: MountPropagation:<nil> SubPathExpr:} {Name:plugin-dir ReadOnly:false MountPath:C:\\csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-config ReadOnly:false MountPath:C:\\k SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-node-win-config ReadOnly:false MountPath:/etc/csi-azuredisk-node-win SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-proxy-fs-pipe-v1 ReadOnly:false MountPath:\\\\.\\pipe\\csi-proxy-filesystem-v1 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-proxy-disk-pipe-v1 ReadOnly:false MountPath:\\\\.\\pipe\\csi-proxy-disk-v1 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-proxy-volume-pipe-v1 ReadOnly:false MountPath:\\\\.\\pipe\\csi-proxy-volume-v1 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-proxy-fs-pipe-v1beta1 ReadOnly:false MountPath:\\\\.\\pipe\\csi-proxy-filesystem-v1beta1 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-proxy-disk-pipe-v1beta2 ReadOnly:false MountPath:\\\\.\\pipe\\csi-proxy-disk-v1beta2 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-proxy-volume-pipe-v1beta2 ReadOnly:false MountPath:\\\\.\\pipe\\csi-proxy-volume-v1beta2 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-dzv4j ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000ca66b8 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:windows] ServiceAccountName:csi-azuredisk-node-sa DeprecatedServiceAccount:csi-azuredisk-node-sa AutomountServiceAccountToken:<nil> NodeName:1417k8s00000000 HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:&Affinity{NodeAffinity:&NodeAffinity{RequiredDuringSchedulingIgnoredDuringExecution:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{},MatchFields:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:metadata.name,Operator:In,Values:[1417k8s00000000],},},},},},PreferredDuringSchedulingIgnoredDuringExecution:[]PreferredSchedulingTerm{},},PodAffinity:nil,PodAntiAffinity:nil,} SchedulerName:default-scheduler Tolerations:[{Key:node.kubernetes.io/os Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:<nil>} {Key:node.kubernetes.io/disk-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/memory-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/pid-pressure Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/unschedulable Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>}] HostAliases:[] PriorityClassName:system-node-critical Priority:0xc000ca66c0 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000ca66c4 PreemptionPolicy:0xc000bd5240 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:csi-proxy-fs-pipe-v1 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-filesystem-v1,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-disk-pipe-v1 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-disk-v1,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-volume-pipe-v1 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-volume-v1,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-node-win-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-node-win-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-fs-pipe-v1beta1 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-filesystem-v1beta1,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-disk-pipe-v1beta2 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-disk-v1beta2,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-proxy-volume-pipe-v1beta2 VolumeSource:{HostPath:&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-volume-v1beta2,Type:*,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:registration-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:C:\\var\\lib\\kubelet\\plugins_registry\\,Type:*Directory,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kubelet-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:C:\\var\\lib\\kubelet\\,Type:*Directory,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:plugin-dir VolumeSource:{HostPath:&HostPathVolumeSource{Path:C:\\var\\lib\\kubelet\\plugins\\disk.csi.azure.com\\,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-config VolumeSource:{HostPath:&HostPathVolumeSource{Path:C:\\k,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-dzv4j VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0920 15:19:02.674508       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-node-win-c49lh: Skipping Volume {csi-proxy-fs-pipe-v1 {&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-filesystem-v1,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0920 15:19:02.674564       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-node-win-c49lh: Skipping Volume {csi-proxy-disk-pipe-v1 {&HostPathVolumeSource{Path:\\\\.\\pipe\\csi-proxy-disk-v1,Type:*,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
... skipping 57 lines ...
I0920 15:19:02.687381       1 shared_state.go:483]  "msg"="Storing pod coredns-7cf78b68d8-xtr22 and claim [] to podToClaimsMap map." 
I0920 15:19:02.687442       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/coredns-7cf78b68d8-xtr22." "disk.csi.azure.com/request-id"="8f13db07-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/coredns-7cf78b68d8-xtr22"
I0920 15:19:02.687488       1 shared_state.go:307]  "msg"="Getting requested volumes for pod (kube-system/coredns-7cf78b68d8-xtr22)." "disk.csi.azure.com/request-id"="8f13db07-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/coredns-7cf78b68d8-xtr22"
I0920 15:19:02.687516       1 pod.go:99]  "msg"="Pod kube-system/coredns-7cf78b68d8-xtr22 has 0 volumes. Volumes: []" "disk.csi.azure.com/request-id"="8f13db07-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/coredns-7cf78b68d8-xtr22"
I0920 15:19:02.687651       1 pod.go:89]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="8f13db07-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcilePod).createReplicas" "disk.csi/azure.com/pod-name"="kube-system/coredns-7cf78b68d8-xtr22" "latency"=201601
I0920 15:19:02.652054       1 shared_state.go:413]  "msg"="Adding pod csi-azuredisk-controller-6f9f9c69fc-4t2ss to shared map with keyName kube-system/csi-azuredisk-controller-6f9f9c69fc-4t2ss." 
I0920 15:19:02.688762       1 shared_state.go:419]  "msg"="Pod spec of pod csi-azuredisk-controller-6f9f9c69fc-4t2ss is: {Volumes:[{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-h4x8f VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:csi-provisioner Image:mcr.microsoft.com/oss/kubernetes-csi/csi-provisioner:v3.2.0 Command:[] Args:[--feature-gates=Topology=true --csi-address=$(ADDRESS) --v=2 --timeout=15s --leader-election --leader-election-namespace=kube-system --worker-threads=40 --extra-create-metadata=true --strict-topology=true] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-attacher Image:mcr.microsoft.com/oss/kubernetes-csi/csi-attacher:v3.5.0 Command:[] Args:[-v=2 -csi-address=$(ADDRESS) -timeout=600s -leader-election --leader-election-namespace=kube-system -worker-threads=500] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-snapshotter Image:mcr.microsoft.com/oss/kubernetes-csi/csi-snapshotter:v5.0.1 Command:[] Args:[-csi-address=$(ADDRESS) -leader-election --leader-election-namespace=kube-system -v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:csi-resizer Image:mcr.microsoft.com/oss/kubernetes-csi/csi-resizer:v1.5.0 Command:[] Args:[-csi-address=$(ADDRESS) -v=2 -leader-election --leader-election-namespace=kube-system -handle-volume-inuse-error=false -feature-gates=RecoverVolumeExpansionFailure=true -timeout=240s] WorkingDir: Ports:[] EnvFrom:[] Env:[{Name:ADDRESS Value:/csi/csi.sock ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:liveness-probe Image:mcr.microsoft.com/oss/kubernetes-csi/livenessprobe:v2.7.0 Command:[] Args:[--csi-address=/csi/csi.sock --probe-timeout=3s --health-port=29602 --v=2] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[memory:{i:{value:104857600 scale:0} d:{Dec:<nil>} s:100Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} {Name:azuredisk Image:k8sprow.azurecr.io/azuredisk-csi:latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 Command:[] Args:[--v=6 --config=/etc/csi-azuredisk-controller/config.yaml] WorkingDir: Ports:[{Name:healthz HostPort:29602 ContainerPort:29602 Protocol:TCP HostIP:} {Name:metrics HostPort:29604 ContainerPort:29604 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:AZURE_CREDENTIAL_FILE Value: ValueFrom:&EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:&ConfigMapKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-cred-file,},Key:path,Optional:*true,},SecretKeyRef:nil,}} {Name:CSI_ENDPOINT Value:unix:///csi/csi.sock ValueFrom:nil} {Name:AZURE_GO_SDK_LOG_LEVEL Value: ValueFrom:nil}] Resources:{Limits:map[memory:{i:{value:524288000 scale:0} d:{Dec:<nil>} s:500Mi Format:BinarySI}] Requests:map[cpu:{i:{value:10 scale:-3} d:{Dec:<nil>} s:10m Format:DecimalSI} memory:{i:{value:20971520 scale:0} d:{Dec:<nil>} s:20Mi Format:BinarySI}]} VolumeMounts:[{Name:socket-dir ReadOnly:false MountPath:/csi SubPath: MountPropagation:<nil> SubPathExpr:} {Name:azure-cred ReadOnly:false MountPath:/etc/kubernetes/ SubPath: MountPropagation:<nil> SubPathExpr:} {Name:csi-azuredisk-controller-config ReadOnly:false MountPath:/etc/csi-azuredisk-controller SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-h4x8f ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:{1 0 healthz},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:10,PeriodSeconds:30,SuccessThreshold:1,FailureThreshold:5,TerminationGracePeriodSeconds:nil,} ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:Always SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc000ca65a0 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:linux] ServiceAccountName:csi-azuredisk-controller-sa DeprecatedServiceAccount:csi-azuredisk-controller-sa AutomountServiceAccountToken:<nil> NodeName:k8s-master-14178977-vmss000000 HostNetwork:true HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName:default-scheduler Tolerations:[{Key:node-role.kubernetes.io/master Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/controlplane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node-role.kubernetes.io/control-plane Operator:Exists Value: Effect:NoSchedule TolerationSeconds:<nil>} {Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000ca65a8} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc000ca65b0}] HostAliases:[] PriorityClassName:system-cluster-critical Priority:0xc000ca65b8 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc000ca65bc PreemptionPolicy:0xc000bd4d30 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:socket-dir VolumeSource:{HostPath:nil EmptyDir:&EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:azure-cred VolumeSource:{HostPath:&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:csi-azuredisk-controller-config VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:&ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-h4x8f VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0920 15:19:02.688856       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6f9f9c69fc-4t2ss: Skipping Volume {socket-dir {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0920 15:19:02.688924       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6f9f9c69fc-4t2ss: Skipping Volume {azure-cred {&HostPathVolumeSource{Path:/etc/kubernetes/,Type:*DirectoryOrCreate,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0920 15:19:02.688999       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6f9f9c69fc-4t2ss: Skipping Volume {csi-azuredisk-controller-config {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:csi-azuredisk-controller-config,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil}}. No persistent volume exists." 
I0920 15:19:02.689103       1 shared_state.go:457]  "msg"="Pod csi-azuredisk-controller-6f9f9c69fc-4t2ss: Skipping Volume {kube-api-access-h4x8f {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0920 15:19:02.689153       1 shared_state.go:483]  "msg"="Storing pod csi-azuredisk-controller-6f9f9c69fc-4t2ss and claim [] to podToClaimsMap map." 
I0920 15:19:02.689206       1 pod.go:91]  "msg"="Creating replicas for pod kube-system/csi-azuredisk-controller-6f9f9c69fc-4t2ss." "disk.csi.azure.com/request-id"="8f142026-38f7-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="kube-system/csi-azuredisk-controller-6f9f9c69fc-4t2ss"
... skipping 276 lines ...
I0920 15:20:26.816571       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-ec43aca1-3abb-4b3c-a901-38a027d61779","parameters":{"csi.storage.k8s.io/pv/name":"pvc-ec43aca1-3abb-4b3c-a901-38a027d61779","csi.storage.k8s.io/pvc/name":"pvc-hq7cf","csi.storage.k8s.io/pvc/namespace":"azuredisk-1884","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0920 15:20:26.817097       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:20:26.831812       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 14 milliseconds
I0920 15:20:26.835021       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:20:26.835060       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-ec43aca1-3abb-4b3c-a901-38a027d61779)
I0920 15:20:26.849711       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779/status 200 OK in 15 milliseconds
I0920 15:20:26.850265       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:20:26.850984       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=17227681
I0920 15:20:26.851016       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:20:26.865117       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:20:26.865305       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.014075966 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:20:26.865342       1 cloudprovisioner.go:246] begin to create disk(pvc-ec43aca1-3abb-4b3c-a901-38a027d61779) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone(westus2-1) maxShares(0)
I0920 15:20:26.882491       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01711338 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:20:26.882531       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-ec43aca1-3abb-4b3c-a901-38a027d61779 StorageAccountType:StandardSSD_LRS Size:10
I0920 15:20:27.003733       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:20:27.013720       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
... skipping 7 lines ...
I0920 15:20:29.186189       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.303597368 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:20:29.206255       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019992694 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:20:29.206298       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-ec43aca1-3abb-4b3c-a901-38a027d61779 StorageAccountType:StandardSSD_LRS Size:10
I0920 15:20:29.206372       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-ec43aca1-3abb-4b3c-a901-38a027d61779) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-ec43aca1-3abb-4b3c-a901-38a027d61779 kubernetes.io-created-for-pvc-name:pvc-hq7cf kubernetes.io-created-for-pvc-namespace:azuredisk-1884]) successfully" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:20:29.206425       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=2355329110
I0920 15:20:29.216885       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779/status 200 OK in 10 milliseconds
I0920 15:20:29.217715       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:20:29.217761       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=2382624038
I0920 15:20:29.217785       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=2400754922
I0920 15:20:29.217819       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.400799422 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" result_code="succeeded"
I0920 15:20:29.217830       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ec43aca1-3abb-4b3c-a901-38a027d61779","csi.storage.k8s.io/pvc/name":"pvc-hq7cf","csi.storage.k8s.io/pvc/namespace":"azuredisk-1884","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"}}
I0920 15:20:29.220932       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=14440867
I0920 15:20:29.220990       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=2369907778
I0920 15:20:29.221034       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=2387283559
I0920 15:20:29.932066       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 15:20:29.932102       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ec43aca1-3abb-4b3c-a901-38a027d61779","csi.storage.k8s.io/pvc/name":"pvc-hq7cf","csi.storage.k8s.io/pvc/namespace":"azuredisk-1884","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"}
I0920 15:20:29.932878       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000000,disk.csi.azure.com/volume-name!=pvc-ec43aca1-3abb-4b3c-a901-38a027d61779." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:20:29.938466       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 5 milliseconds
I0920 15:20:29.939452       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment)
I0920 15:20:29.942925       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:20:29.950695       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment/status 200 OK in 7 milliseconds
I0920 15:20:29.952808       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:20:29.953751       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=10754151
I0920 15:20:29.953803       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:20:29.978781       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.024847116 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:20:29.978853       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:20:29.996569       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017451082 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:20:29.996598       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
... skipping 7 lines ...
I0920 15:20:31.307329       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-y9dto3pe|1417k8s00000000" delay="1s"
I0920 15:20:31.307440       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:20:31.323342       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015842574 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:20:31.323373       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:20:31.323590       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:20:31.335244       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment/status 200 OK in 11 milliseconds
I0920 15:20:31.337190       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:20:31.337516       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=1397760634
I0920 15:20:31.337725       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=1398214635
I0920 15:20:31.337788       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=1405158968
I0920 15:20:31.337880       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.405280768 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" node="1417k8s00000000" result_code="succeeded"
I0920 15:20:31.338241       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=14347267
I0920 15:20:31.337904       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
... skipping 2 lines ...
I0920 15:20:31.347595       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779:AttachDiskOptions{diskName: "pvc-ec43aca1-3abb-4b3c-a901-38a027d61779", lun: 0}]
I0920 15:20:31.347647       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779:AttachDiskOptions{diskName: "pvc-ec43aca1-3abb-4b3c-a901-38a027d61779", lun: 0}])
I0920 15:20:31.821542       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:20:31.826947       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:20:31.827142       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:20:31.921350       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.57333868 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 15:20:31.921392       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779:AttachDiskOptions{diskName: "pvc-ec43aca1-3abb-4b3c-a901-38a027d61779", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 15:20:32.362951       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:20:32.362950       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:20:32.362973       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:20:33.050646       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:20:33.054594       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:20:33.055300       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 172 lines ...
I0920 15:21:56.140906       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"}
I0920 15:21:56.141678       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment) detachment" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:21:56.153565       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment/status 200 OK in 11 milliseconds
I0920 15:21:56.154061       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=12303813
I0920 15:21:56.161199       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment 200 OK in 6 milliseconds
I0920 15:21:56.162014       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment)
I0920 15:21:56.165138       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:21:56.166209       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-ec43aca1-3abb-4b3c-a901-38a027d61779) scheduled in 5m0s." "disk.csi.azure.com/request-id"="f67a9b63-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:21:56.175219       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:21:56.177419       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment/status 200 OK in 11 milliseconds
I0920 15:21:56.177936       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=12098114
I0920 15:21:56.177980       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:21:56.178117       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779 from node 1417k8s00000000" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:21:56.178243       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:21:56.195084       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016698019 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 19 lines ...
I0920 15:22:01.561794       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:22:01.566553       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:22:01.566752       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:22:01.822697       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:22:01.828851       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:22:01.828956       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:22:01.829046       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:22:02.364191       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:22:02.365419       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:22:02.366503       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:22:03.570784       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:22:03.574629       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:22:03.575037       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 22 lines ...
I0920 15:22:17.632333       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779 from node 1417k8s00000000 successfully" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:22:17.632604       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=21454520314
I0920 15:22:17.647884       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment 200 OK in 14 milliseconds
I0920 15:22:17.649090       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=16222619
I0920 15:22:17.649162       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=21471112733
I0920 15:22:17.649201       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=21483392548
I0920 15:22:17.652661       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:22:17.652742       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=21490544656
I0920 15:22:17.652805       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c3146acc-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=21490841356
I0920 15:22:17.652999       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="f676dee1-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=21511275178
I0920 15:22:17.653113       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=21.511672479 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" node="1417k8s00000000" result_code="succeeded"
I0920 15:22:17.653186       1 utils.go:85] GRPC response: {}
I0920 15:22:17.653925       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
... skipping 59 lines ...
I0920 15:22:46.427093       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0920 15:22:46.427120       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"}
I0920 15:22:46.427406       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779)
I0920 15:22:46.450739       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779/status 200 OK in 22 milliseconds
I0920 15:22:46.453628       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="146fe5b3-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=25974672
I0920 15:22:46.453654       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-ec43aca1-3abb-4b3c-a901-38a027d61779)
I0920 15:22:46.463361       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:22:46.466204       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779 200 OK in 12 milliseconds
I0920 15:22:46.475966       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779/status 200 OK in 10 milliseconds
I0920 15:22:46.476731       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=11244331
I0920 15:22:46.477031       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:22:46.477382       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-ec43aca1-3abb-4b3c-a901-38a027d61779)" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:22:46.478003       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-ec43aca1-3abb-4b3c-a901-38a027d61779)" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:22:46.481826       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:22:46.482638       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-ec43aca1-3abb-4b3c-a901-38a027d61779." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"
I0920 15:22:46.482707       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=5346915
I0920 15:22:46.511999       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02920688 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:22:47.835183       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:22:47.839162       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:22:47.839933       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:22:49.843404       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0920 15:22:49.849286       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:22:49.849456       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:22:51.676382       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.164234955 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:22:51.676547       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779
I0920 15:22:51.676831       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=5193820636
I0920 15:22:51.700190       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779 200 OK in 22 milliseconds
I0920 15:22:51.702557       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:22:51.702648       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="146fe5b3-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=5236018885
I0920 15:22:51.702707       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="146fe5b3-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=5275086992
I0920 15:22:51.702724       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779) returned with <nil>
I0920 15:22:51.702777       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.275334393 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" result_code="succeeded"
I0920 15:22:51.702847       1 utils.go:85] GRPC response: {}
I0920 15:22:51.706463       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "disk.csi.azure.com/request-id"="c13903c8-38f7-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=29423017
... skipping 12 lines ...
I0920 15:22:55.517967       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 15:22:55.518216       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-99dc0e3e-8916-4ec1-a708-98f550212e26","parameters":{"csi.storage.k8s.io/pv/name":"pvc-99dc0e3e-8916-4ec1-a708-98f550212e26","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-kxfpl-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-4933","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0920 15:22:55.518865       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:22:55.561683       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 42 milliseconds
I0920 15:22:55.567899       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:22:55.568160       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26)
I0920 15:22:55.597698       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:22:55.602855       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26/status 200 OK in 39 milliseconds
I0920 15:22:55.604915       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=42023668
I0920 15:22:55.604950       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:22:55.638170       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:22:55.638397       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.033190533 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:22:55.638424       1 cloudprovisioner.go:246] begin to create disk(pvc-99dc0e3e-8916-4ec1-a708-98f550212e26) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0920 15:22:55.704385       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.065820363 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:22:55.704431       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-99dc0e3e-8916-4ec1-a708-98f550212e26 StorageAccountType:StandardSSD_LRS Size:10
I0920 15:22:55.880718       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:22:55.885640       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 8 lines ...
I0920 15:22:57.922129       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020318813 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:22:57.922163       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-99dc0e3e-8916-4ec1-a708-98f550212e26 StorageAccountType:StandardSSD_LRS Size:10
I0920 15:22:57.922484       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-99dc0e3e-8916-4ec1-a708-98f550212e26) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-99dc0e3e-8916-4ec1-a708-98f550212e26 kubernetes.io-created-for-pvc-name:test-volume-1-azuredisk-volume-tester-kxfpl-0 kubernetes.io-created-for-pvc-namespace:azuredisk-4933]) successfully" "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:22:57.922790       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=2317503023
I0920 15:22:57.938251       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26/status 200 OK in 15 milliseconds
I0920 15:22:57.941523       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=18649203
I0920 15:22:57.942001       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:22:57.942079       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=2373598878
I0920 15:22:57.942119       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=2423302577
I0920 15:22:57.942170       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=2336575128
I0920 15:22:57.942219       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=2379346700
I0920 15:22:57.942455       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.423360477 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" result_code="succeeded"
I0920 15:22:57.942484       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-99dc0e3e-8916-4ec1-a708-98f550212e26","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-kxfpl-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-4933","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"}}
I0920 15:22:58.591260       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 15:22:58.591503       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-99dc0e3e-8916-4ec1-a708-98f550212e26","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-kxfpl-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-4933","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"}
I0920 15:22:58.592018       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000001,disk.csi.azure.com/volume-name!=pvc-99dc0e3e-8916-4ec1-a708-98f550212e26." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:22:58.599938       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 7 milliseconds
I0920 15:22:58.601540       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000001-attachment)
I0920 15:22:58.622133       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:22:58.623049       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000001-attachment/status 200 OK in 22 milliseconds
I0920 15:22:58.623598       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=22930926
I0920 15:22:58.623655       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:22:58.645323       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021490418 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:22:58.645623       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:22:58.663044       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017114494 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 2 lines ...
I0920 15:22:58.663272       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26\" to node \"1417k8s00000001\"." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:22:59.664260       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-y9dto3pe|1417k8s00000001" delay="1s"
I0920 15:22:59.664385       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:22:59.722654       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.058175021 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:22:59.722711       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:22:59.722788       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:22:59.733226       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:22:59.733302       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=1131670150
I0920 15:22:59.733347       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=1131825650
I0920 15:22:59.733383       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=1141671606
I0920 15:22:59.733426       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.141750606 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" node="1417k8s00000001" result_code="succeeded"
I0920 15:22:59.733444       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 15:22:59.737767       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.014942083 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:22:59.737836       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:22:59.737854       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26:AttachDiskOptions{diskName: "pvc-99dc0e3e-8916-4ec1-a708-98f550212e26", lun: 0}]
I0920 15:22:59.737955       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26:AttachDiskOptions{diskName: "pvc-99dc0e3e-8916-4ec1-a708-98f550212e26", lun: 0}])
I0920 15:22:59.739779       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000001-attachment/status 200 OK in 16 milliseconds
I0920 15:22:59.740550       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=17629097
I0920 15:22:59.860729       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.122621478 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 15:22:59.860794       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26:AttachDiskOptions{diskName: "pvc-99dc0e3e-8916-4ec1-a708-98f550212e26", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 15:22:59.899893       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:22:59.905185       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:22:59.906847       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:23:01.823249       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:23:01.829418       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:23:01.829425       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 123 lines ...
I0920 15:23:53.729579       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000001-attachment)
I0920 15:23:53.739647       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000001-attachment/status 200 OK in 13 milliseconds
I0920 15:23:53.742260       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16960050
I0920 15:23:53.742542       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:23:53.742814       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26 from node 1417k8s00000001" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:23:53.743051       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:23:53.745058       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:23:53.760438       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01719255 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:23:53.760462       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:23:53.760611       1 azure_vmss.go:204] Couldn't find VMSS VM with nodeName 1417k8s00000001, refreshing the cache(vmss: 1417k8s00, rg: kubetest-y9dto3pe)
I0920 15:23:53.844027       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.083369944 request="vmssvm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:23:54.251596       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:23:54.255365       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
... skipping 17 lines ...
I0920 15:24:00.288100       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:24:00.292631       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:24:00.292932       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:24:01.825382       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:24:01.831593       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:24:01.831605       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:24:01.831662       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:24:02.300351       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:24:02.306840       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:24:02.307355       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:24:02.366653       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:24:02.366744       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:24:02.368838       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 15 lines ...
I0920 15:24:10.131470       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26 from node 1417k8s00000001 successfully" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:24:10.131527       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16388745539
I0920 15:24:10.145127       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000001-attachment 200 OK in 13 milliseconds
I0920 15:24:10.145901       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=14311636
I0920 15:24:10.145947       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16403183477
I0920 15:24:10.145980       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16420709029
I0920 15:24:10.149089       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:24:10.149181       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16419324923
I0920 15:24:10.149246       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1baffdd6-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16419681024
I0920 15:24:10.149295       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="3c86c57b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16462904050
I0920 15:24:10.149352       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.46302465 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" node="1417k8s00000001" result_code="succeeded"
I0920 15:24:10.149380       1 utils.go:85] GRPC response: {}
I0920 15:24:10.349344       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
... skipping 402 lines ...
I0920 15:27:40.162478       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000003","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-99dc0e3e-8916-4ec1-a708-98f550212e26","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-kxfpl-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-4933","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"}
I0920 15:27:40.163061       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000003,disk.csi.azure.com/volume-name!=pvc-99dc0e3e-8916-4ec1-a708-98f550212e26." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:27:40.173849       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 10 milliseconds
I0920 15:27:40.177210       1 replica.go:157]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="3c8cbb55-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileReplica).triggerGarbageCollection" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=226451756966
I0920 15:27:40.177695       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000003-attachment)
I0920 15:27:40.188761       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000003-attachment/status 200 OK in 11 milliseconds
I0920 15:27:40.190303       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:27:40.191546       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=14825627
I0920 15:27:40.191817       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:27:40.240995       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.048777187 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:27:40.241070       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:27:40.269881       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028767451 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:27:40.269920       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
... skipping 3 lines ...
I0920 15:27:41.270555       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:27:41.290286       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019685435 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:27:41.290317       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:27:41.290369       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:27:41.301138       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000003-attachment/status 200 OK in 10 milliseconds
I0920 15:27:41.301686       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=11023920
I0920 15:27:41.303573       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:27:41.303943       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=1126049323
I0920 15:27:41.304260       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=1126592123
I0920 15:27:41.304724       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=1141743551
I0920 15:27:41.304995       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.142077552 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" node="1417k8s00000003" result_code="succeeded"
I0920 15:27:41.305285       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 15:27:41.310092       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019648436 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 8 lines ...
I0920 15:27:41.314171       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.000439401 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" node="1417k8s00000003" result_code="succeeded"
I0920 15:27:41.314186       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 15:27:41.695807       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:27:41.700795       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:27:41.701150       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:27:42.791822       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.480163358 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 15:27:42.791860       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26:AttachDiskOptions{diskName: "pvc-99dc0e3e-8916-4ec1-a708-98f550212e26", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 15:27:43.705762       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:27:43.709729       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:27:43.710327       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:27:45.717796       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:27:45.722378       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:27:45.722788       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 121 lines ...
I0920 15:28:34.245688       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000003-attachment) detachment" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:28:34.254372       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000003-attachment/status 200 OK in 8 milliseconds
I0920 15:28:34.256533       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=10778490
I0920 15:28:34.263672       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000003-attachment 200 OK in 6 milliseconds
I0920 15:28:34.265949       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000003-attachment)
I0920 15:28:34.269203       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26) scheduled in 5m0s." "disk.csi.azure.com/request-id"="e3c44eae-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:28:34.269572       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:28:34.277774       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:28:34.279329       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000003-attachment/status 200 OK in 10 milliseconds
I0920 15:28:34.279957       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=11227190
I0920 15:28:34.280035       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:28:34.280543       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26 from node 1417k8s00000003" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:28:34.280580       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:28:34.300675       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020042182 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 40 lines ...
I0920 15:28:50.702189       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=15.279398545 request="vmssvm_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="detach_disk" error_code=""
I0920 15:28:50.702415       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000003) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26:pvc-99dc0e3e-8916-4ec1-a708-98f550212e26]) returned with <nil>
I0920 15:28:50.702622       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 1417k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26:pvc-99dc0e3e-8916-4ec1-a708-98f550212e26]
I0920 15:28:50.702876       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-99dc0e3e-8916-4ec1-a708-98f550212e26, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26) succeeded
I0920 15:28:50.703080       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26 from node 1417k8s00000003 successfully" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:28:50.703259       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16422846804
I0920 15:28:50.717027       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:28:50.717518       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16450429502
I0920 15:28:50.717820       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="c38459ee-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16451819702
I0920 15:28:50.718140       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="e3c0b6f2-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=16472414584
I0920 15:28:50.718804       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.472822084 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" node="1417k8s00000003" result_code="succeeded"
I0920 15:28:50.719016       1 utils.go:85] GRPC response: {}
I0920 15:28:50.719884       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26-1417k8s00000003-attachment 200 OK in 16 milliseconds
... skipping 19 lines ...
I0920 15:28:58.721134       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"}
I0920 15:28:58.721367       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26)
I0920 15:28:58.739117       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26/status 200 OK in 17 milliseconds
I0920 15:28:58.739492       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="f2577373-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=17837717
I0920 15:28:58.739532       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26)
I0920 15:28:58.748361       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26 200 OK in 8 milliseconds
I0920 15:28:58.749948       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:28:58.759696       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26/status 200 OK in 7 milliseconds
I0920 15:28:58.761401       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:28:58.763680       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=11988111
I0920 15:28:58.763788       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:28:58.764006       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26)" "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:28:58.764118       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-99dc0e3e-8916-4ec1-a708-98f550212e26)" "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:28:58.764245       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-99dc0e3e-8916-4ec1-a708-98f550212e26." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26"
I0920 15:28:58.764542       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=493500
I0920 15:28:58.799022       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.034062432 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:00.207977       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:29:00.213245       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:29:00.213575       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:29:01.834566       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:29:01.840790       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:29:01.840953       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:01.840806       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:29:02.220640       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:29:02.225308       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:29:02.225658       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:29:02.374165       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:29:02.374286       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:29:02.377585       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:29:03.964671       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.165493232 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:03.964705       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26
I0920 15:29:03.964806       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="19db1afa-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=5199858764
I0920 15:29:03.981075       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:29:03.981461       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="f2577373-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=5232431595
I0920 15:29:03.981735       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "disk.csi.azure.com/request-id"="f2577373-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" "latency"=5259912321
I0920 15:29:03.981948       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26) returned with <nil>
I0920 15:29:03.982186       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.260792821 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26" result_code="succeeded"
I0920 15:29:03.982402       1 utils.go:85] GRPC response: {}
I0920 15:29:03.983502       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-99dc0e3e-8916-4ec1-a708-98f550212e26 200 OK in 18 milliseconds
... skipping 30 lines ...
I0920 15:29:14.243868       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb)
I0920 15:29:14.260912       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 15:29:14.261196       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f","csi.storage.k8s.io/pvc/name":"pvc-mvz7j","csi.storage.k8s.io/pvc/namespace":"azuredisk-9164","skuname":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0920 15:29:14.261870       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:29:14.265769       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 36 milliseconds
I0920 15:29:14.267371       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb/status 200 OK in 20 milliseconds
I0920 15:29:14.269840       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:14.271334       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:29:14.271537       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-8d5af7b9-8401-45b6-af24-a67592ffc076)
I0920 15:29:14.274164       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=28886435
I0920 15:29:14.274219       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:29:14.279234       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076/status 200 OK in 13 milliseconds
I0920 15:29:14.282722       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=17306021
I0920 15:29:14.282780       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:29:14.286215       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:14.286989       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 24 milliseconds
I0920 15:29:14.288208       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:29:14.288253       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f)
I0920 15:29:14.293870       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:29:14.294280       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019924024 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:29:14.294320       1 cloudprovisioner.go:246] begin to create disk(pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb) account type(Standard_LRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0920 15:29:14.308563       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:14.308734       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:14.309877       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 13 milliseconds
I0920 15:29:14.310127       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:29:14.310347       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.027307533 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:29:14.310471       1 cloudprovisioner.go:246] begin to create disk(pvc-8d5af7b9-8401-45b6-af24-a67592ffc076) account type(Premium_LRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0920 15:29:14.311625       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f/status 200 OK in 21 milliseconds
I0920 15:29:14.312061       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=22001426
I0920 15:29:14.312206       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:29:14.316052       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:29:14.316521       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:29:14.329379       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:29:14.329575       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01685972 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:29:14.329717       1 cloudprovisioner.go:246] begin to create disk(pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0920 15:29:14.355847       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.045258754 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:14.355891       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-8d5af7b9-8401-45b6-af24-a67592ffc076 StorageAccountType:Premium_LRS Size:10
I0920 15:29:14.358433       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.064080476 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:14.358464       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb StorageAccountType:Standard_LRS Size:10
... skipping 9 lines ...
I0920 15:29:16.650892       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f kubernetes.io-created-for-pvc-name:pvc-mvz7j kubernetes.io-created-for-pvc-namespace:azuredisk-9164]) successfully" "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:29:16.650939       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=2338443183
I0920 15:29:16.652012       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023817128 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:16.652234       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-8d5af7b9-8401-45b6-af24-a67592ffc076 StorageAccountType:Premium_LRS Size:10
I0920 15:29:16.652530       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-8d5af7b9-8401-45b6-af24-a67592ffc076) account type(Premium_LRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-8d5af7b9-8401-45b6-af24-a67592ffc076 kubernetes.io-created-for-pvc-name:pvc-xncdd kubernetes.io-created-for-pvc-namespace:azuredisk-9164]) successfully" "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:29:16.652720       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=2369751420
I0920 15:29:16.670546       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:29:16.670685       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=2382248034
I0920 15:29:16.670804       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=2408995566
I0920 15:29:16.671007       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.409230266 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" result_code="succeeded"
I0920 15:29:16.671103       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f","csi.storage.k8s.io/pvc/name":"pvc-mvz7j","csi.storage.k8s.io/pvc/namespace":"azuredisk-9164","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"}}
I0920 15:29:16.678347       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f/status 200 OK in 26 milliseconds
I0920 15:29:16.682098       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076/status 200 OK in 28 milliseconds
I0920 15:29:16.683185       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=30219736
I0920 15:29:16.683474       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=2400567056
I0920 15:29:16.683763       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=2418363778
I0920 15:29:16.685134       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:29:16.685796       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=34738541
I0920 15:29:16.686219       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=2373800025
I0920 15:29:16.686514       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=2396473151
I0920 15:29:16.686779       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=2414888873
I0920 15:29:16.687014       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=2458835326
I0920 15:29:16.687302       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.459135526 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" result_code="succeeded"
... skipping 4 lines ...
I0920 15:29:16.724541       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb) account type(Standard_LRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb kubernetes.io-created-for-pvc-name:pvc-svjzs kubernetes.io-created-for-pvc-namespace:azuredisk-9164]) successfully" "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:29:16.724748       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=2450442715
I0920 15:29:16.740007       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb/status 200 OK in 14 milliseconds
I0920 15:29:16.749541       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=24559729
I0920 15:29:16.749639       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=2475341045
I0920 15:29:16.749674       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=2504444480
I0920 15:29:16.750377       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:29:16.750962       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=2507015883
I0920 15:29:16.751026       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=2547535231
I0920 15:29:16.751078       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.5476110309999997 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" result_code="succeeded"
I0920 15:29:16.751108       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb","csi.storage.k8s.io/pvc/name":"pvc-svjzs","csi.storage.k8s.io/pvc/namespace":"azuredisk-9164","requestedsizegib":"10","skuname":"Standard_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"}}
I0920 15:29:17.379558       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 15:29:17.379850       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f","csi.storage.k8s.io/pvc/name":"pvc-mvz7j","csi.storage.k8s.io/pvc/namespace":"azuredisk-9164","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"}
... skipping 5 lines ...
I0920 15:29:17.391126       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb","csi.storage.k8s.io/pvc/name":"pvc-svjzs","csi.storage.k8s.io/pvc/namespace":"azuredisk-9164","requestedsizegib":"10","skuname":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"}
I0920 15:29:17.391615       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000001,disk.csi.azure.com/volume-name!=pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:29:17.399706       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 14 milliseconds
I0920 15:29:17.402821       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 21 milliseconds
I0920 15:29:17.403415       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8d5af7b9-8401-45b6-af24-a67592ffc076-1417k8s00000001-attachment)
I0920 15:29:17.403969       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f-1417k8s00000001-attachment)
I0920 15:29:17.411529       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:17.412101       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 20 milliseconds
I0920 15:29:17.415316       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:17.423993       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb-1417k8s00000001-attachment)
I0920 15:29:17.426348       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:17.431994       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076-1417k8s00000001-attachment/status 200 OK in 17 milliseconds
I0920 15:29:17.432162       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f-1417k8s00000001-attachment/status 200 OK in 21 milliseconds
I0920 15:29:17.432913       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=22843627
I0920 15:29:17.433609       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:29:17.434880       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:17.436554       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=22105126
I0920 15:29:17.436762       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:29:17.442980       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:17.444148       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb-1417k8s00000001-attachment/status 200 OK in 18 milliseconds
I0920 15:29:17.444822       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=19810023
I0920 15:29:17.454172       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:29:17.455470       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021579726 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:17.455537       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:29:17.456170       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:29:17.462309       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02520083 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:17.475847       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021535725 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:17.487747       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.032172038 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:17.487774       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:29:17.487971       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:29:17.488225       1 cloudprovisioner.go:398]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f\" to node \"1417k8s00000001\"." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
... skipping 13 lines ...
I0920 15:29:18.488833       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-y9dto3pe|1417k8s00000001" delay="1s"
I0920 15:29:18.488932       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:29:18.505758       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01678332 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:18.505846       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:29:18.505968       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:29:18.526981       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f-1417k8s00000001-attachment/status 200 OK in 20 milliseconds
I0920 15:29:18.528268       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:29:18.528928       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=1122473535
I0920 15:29:18.529122       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=1125050338
I0920 15:29:18.529250       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=1148798966
I0920 15:29:18.529414       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.149016467 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" node="1417k8s00000001" result_code="succeeded"
I0920 15:29:18.529500       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 15:29:18.531488       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076-1417k8s00000001-attachment/status 200 OK in 24 milliseconds
I0920 15:29:18.541739       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=35457742
I0920 15:29:18.542374       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=35569642
I0920 15:29:18.543892       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb-1417k8s00000001-attachment/status 200 OK in 36 milliseconds
I0920 15:29:18.544427       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:29:18.544647       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:29:18.544427       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=37311145
I0920 15:29:18.545182       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=1141056857
I0920 15:29:18.545452       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=1142013759
I0920 15:29:18.545728       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=1161331882
I0920 15:29:18.545952       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.161662482 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" node="1417k8s00000001" result_code="succeeded"
I0920 15:29:18.546206       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
... skipping 11 lines ...
I0920 15:29:18.559985       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0920 15:29:18.579710       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.073580988 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:29:18.579744       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:29:18.579761       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb:AttachDiskOptions{diskName: "pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076:AttachDiskOptions{diskName: "pvc-8d5af7b9-8401-45b6-af24-a67592ffc076", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f:AttachDiskOptions{diskName: "pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f", lun: 0}]
I0920 15:29:18.579839       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb:AttachDiskOptions{diskName: "pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076:AttachDiskOptions{diskName: "pvc-8d5af7b9-8401-45b6-af24-a67592ffc076", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f:AttachDiskOptions{diskName: "pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f", lun: 0}])
I0920 15:29:18.786629       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.206716546 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 15:29:18.786664       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb:AttachDiskOptions{diskName: "pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076:AttachDiskOptions{diskName: "pvc-8d5af7b9-8401-45b6-af24-a67592ffc076", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f:AttachDiskOptions{diskName: "pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 15:29:20.347303       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:29:20.351470       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 15:29:20.351874       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:29:22.360189       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 15:29:22.365374       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:29:22.366058       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 175 lines ...
I0920 15:30:21.908746       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f-1417k8s00000001-attachment 200 OK in 23 milliseconds
I0920 15:30:21.914803       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f-1417k8s00000001-attachment)
I0920 15:30:21.915074       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f) scheduled in 5m0s." "disk.csi.azure.com/request-id"="23edc3d5-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:30:21.925888       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb-1417k8s00000001-attachment/status 200 OK in 19 milliseconds
I0920 15:30:21.926764       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=20512624
I0920 15:30:21.932746       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f-1417k8s00000001-attachment/status 200 OK in 18 milliseconds
I0920 15:30:21.935230       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:21.937319       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=23810829
I0920 15:30:21.937680       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:30:21.937911       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f from node 1417k8s00000001" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:30:21.937938       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:30:21.938423       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0920 15:30:21.938443       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000001","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"}
... skipping 4 lines ...
I0920 15:30:21.953864       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076-1417k8s00000001-attachment/status 200 OK in 14 milliseconds
I0920 15:30:21.962634       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=23739128
I0920 15:30:21.964139       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.026145431 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:30:21.964561       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:30:21.968345       1 azure_vmss.go:204] Couldn't find VMSS VM with nodeName 1417k8s00000001, refreshing the cache(vmss: 1417k8s00, rg: kubetest-y9dto3pe)
I0920 15:30:21.977304       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb-1417k8s00000001-attachment/status 200 OK in 29 milliseconds
I0920 15:30:21.978120       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:21.979786       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=31891038
I0920 15:30:21.979814       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:30:21.979985       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb from node 1417k8s00000001" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:30:21.980006       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:30:21.985093       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076-1417k8s00000001-attachment 200 OK in 20 milliseconds
I0920 15:30:21.985648       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8d5af7b9-8401-45b6-af24-a67592ffc076-1417k8s00000001-attachment)
I0920 15:30:21.986707       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-8d5af7b9-8401-45b6-af24-a67592ffc076) scheduled in 5m0s." "disk.csi.azure.com/request-id"="23f8b08e-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:30:21.989441       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:21.994979       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:21.997088       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076-1417k8s00000001-attachment/status 200 OK in 10 milliseconds
I0920 15:30:21.997650       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=11036113
I0920 15:30:21.997753       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:30:21.997980       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076 from node 1417k8s00000001" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:30:22.005640       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02558073 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:30:22.005672       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
... skipping 28 lines ...
I0920 15:30:30.854397       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:30:30.858966       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:30:30.859098       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:30:31.836459       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:30:31.843631       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:30:31.843632       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:30:31.844066       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:31.844370       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:31.844563       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:32.376816       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:30:32.376816       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:30:32.378997       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:30:32.868462       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 15:30:32.875762       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:30:32.876685       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 14 lines ...
I0920 15:30:38.366775       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb) succeeded
I0920 15:30:38.366801       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb from node 1417k8s00000001 successfully" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:30:38.366835       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=16386925592
I0920 15:30:38.366765       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=16368906271
I0920 15:30:38.387506       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f-1417k8s00000001-attachment 200 OK in 20 milliseconds
I0920 15:30:38.389715       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb-1417k8s00000001-attachment 200 OK in 22 milliseconds
I0920 15:30:38.391491       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:30:38.391763       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=16476703900
I0920 15:30:38.392815       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=16478033901
I0920 15:30:38.393126       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="23e68bf3-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=16525364658
I0920 15:30:38.393414       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.525977658 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" node="1417k8s00000001" result_code="succeeded"
I0920 15:30:38.393668       1 utils.go:85] GRPC response: {}
I0920 15:30:38.396447       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=29561236
I0920 15:30:38.396726       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=16416821229
I0920 15:30:38.396953       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=16449105667
I0920 15:30:38.398305       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:30:38.398605       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=16446803164
I0920 15:30:38.398607       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=31790939
I0920 15:30:38.399218       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=16461429182
I0920 15:30:38.399322       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd768e3b-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=16485893911
I0920 15:30:38.399117       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd783fdd-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=16447439065
I0920 15:30:38.399431       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="23ec5476-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=16493726920
I0920 15:30:38.399512       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.49398152 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" node="1417k8s00000001" result_code="succeeded"
I0920 15:30:38.399530       1 utils.go:85] GRPC response: {}
I0920 15:30:38.401829       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076-1417k8s00000001-attachment 200 OK in 34 milliseconds
I0920 15:30:38.405568       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=38378347
I0920 15:30:38.405872       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=16407964919
I0920 15:30:38.407555       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=16419532632
I0920 15:30:38.414611       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:30:38.415029       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=16428406243
I0920 15:30:38.415524       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="fd7728f7-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=16429764545
I0920 15:30:38.415835       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="23f16320-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=16476965501
I0920 15:30:38.416185       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.477364902 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" node="1417k8s00000001" result_code="succeeded"
I0920 15:30:38.416455       1 utils.go:85] GRPC response: {}
I0920 15:30:38.426897       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 21 lines ...
I0920 15:30:43.277325       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"}
I0920 15:30:43.277622       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f)
I0920 15:30:43.291563       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f/status 200 OK in 13 milliseconds
I0920 15:30:43.292417       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="30a97b9b-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=14449118
I0920 15:30:43.292444       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f)
I0920 15:30:43.300057       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f 200 OK in 7 milliseconds
I0920 15:30:43.302073       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:43.310904       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f/status 200 OK in 7 milliseconds
I0920 15:30:43.313340       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=10419212
I0920 15:30:43.313592       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:30:43.314990       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f)" "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:30:43.315067       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f)" "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:30:43.315166       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f"
I0920 15:30:43.315285       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=337001
I0920 15:30:43.316712       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:43.338467       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022962828 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:30:44.966123       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:30:44.972626       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:30:44.973140       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:30:46.980437       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:30:46.986391       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:30:46.986779       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:30:48.527988       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.189438546 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:30:48.528022       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f
I0920 15:30:48.528081       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="fb9ab32a-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=5212573674
I0920 15:30:48.541315       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:30:48.541578       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="30a97b9b-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=5240827809
I0920 15:30:48.541722       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "disk.csi.azure.com/request-id"="30a97b9b-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" "latency"=5263769137
I0920 15:30:48.541811       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f) returned with <nil>
I0920 15:30:48.541916       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.264219937 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f" result_code="succeeded"
I0920 15:30:48.541948       1 utils.go:85] GRPC response: {}
I0920 15:30:48.543285       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-a40e4957-0b56-4cb2-bd20-0d95904f587f 200 OK in 14 milliseconds
... skipping 18 lines ...
I0920 15:30:53.151844       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"}
I0920 15:30:53.151931       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076)
I0920 15:30:53.172886       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076/status 200 OK in 20 milliseconds
I0920 15:30:53.177452       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="368c2603-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=25373532
I0920 15:30:53.177503       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-8d5af7b9-8401-45b6-af24-a67592ffc076)
I0920 15:30:53.186947       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076 200 OK in 9 milliseconds
I0920 15:30:53.191408       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:53.200516       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076/status 200 OK in 10 milliseconds
I0920 15:30:53.202889       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=13272017
I0920 15:30:53.203085       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:30:53.203242       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-8d5af7b9-8401-45b6-af24-a67592ffc076)" "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:30:53.203273       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-8d5af7b9-8401-45b6-af24-a67592ffc076)" "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:30:53.203334       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-8d5af7b9-8401-45b6-af24-a67592ffc076." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076"
I0920 15:30:53.203395       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=154800
I0920 15:30:53.209051       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:30:53.227077       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02360393 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:30:55.084388       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:30:55.090243       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:30:55.090484       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:30:57.094230       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0920 15:30:57.102916       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 15:30:57.103373       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:30:57.760620       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0920 15:30:57.761014       1 utils.go:79] GRPC request: {}
I0920 15:30:57.761073       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0920 15:30:58.368370       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.141193939 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:30:58.368427       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076
I0920 15:30:58.368589       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="fb959122-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=5165012369
I0920 15:30:58.383669       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:30:58.384735       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="368c2603-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=5192919508
I0920 15:30:58.384990       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "disk.csi.azure.com/request-id"="368c2603-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" "latency"=5232959658
I0920 15:30:58.385184       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076) returned with <nil>
I0920 15:30:58.385391       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.233429758 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076" result_code="succeeded"
I0920 15:30:58.385586       1 utils.go:85] GRPC response: {}
I0920 15:30:58.388930       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d5af7b9-8401-45b6-af24-a67592ffc076 200 OK in 19 milliseconds
... skipping 23 lines ...
I0920 15:31:03.584372       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0920 15:31:03.584406       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"}
I0920 15:31:03.584756       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb)
I0920 15:31:03.594626       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb/status 200 OK in 9 milliseconds
I0920 15:31:03.595283       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="3cc416c8-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=10275014
I0920 15:31:03.595361       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb)
I0920 15:31:03.595818       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:03.605455       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb 200 OK in 9 milliseconds
I0920 15:31:03.607332       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:03.617495       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:03.620732       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb/status 200 OK in 11 milliseconds
I0920 15:31:03.621321       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=11892916
I0920 15:31:03.621515       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:31:03.621751       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb)" "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:31:03.621806       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb)" "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
I0920 15:31:03.621962       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb"
... skipping 11 lines ...
I0920 15:31:08.811518       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb
I0920 15:31:08.814818       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=5189349077
I0920 15:31:08.832072       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb 200 OK in 16 milliseconds
I0920 15:31:08.833104       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=18205823
I0920 15:31:08.833368       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=5211619506
I0920 15:31:08.833664       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="fb91cc8f-38f8-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=5224259724
I0920 15:31:08.838480       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:08.838588       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="3cc416c8-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=5229767030
I0920 15:31:08.838670       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "disk.csi.azure.com/request-id"="3cc416c8-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" "latency"=5253667564
I0920 15:31:08.838699       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb) returned with <nil>
I0920 15:31:08.838773       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.253963666 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-7606ae95-cf59-4bc8-bef8-9ef1cd55cffb" result_code="succeeded"
I0920 15:31:08.838835       1 utils.go:85] GRPC response: {}
I0920 15:31:09.172003       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
... skipping 20 lines ...
I0920 15:31:14.941268       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-d94df931-324b-4db3-ac29-91a858763a58)
I0920 15:31:14.941225       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 15:31:14.942140       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64","parameters":{"csi.storage.k8s.io/pv/name":"pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-6rkjt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","skuname":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0920 15:31:14.942670       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:31:14.957929       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:31:14.958222       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b)
I0920 15:31:14.961670       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:14.965245       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 15:31:14.965278       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3","parameters":{"csi.storage.k8s.io/pv/name":"pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-6rkjt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","skuname":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0920 15:31:14.965595       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:31:14.989369       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-d94df931-324b-4db3-ac29-91a858763a58/status 200 OK in 28 milliseconds
I0920 15:31:14.995017       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 15:31:14.995052       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2","parameters":{"csi.storage.k8s.io/pv/name":"pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-wftzv-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","skuname":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
... skipping 8 lines ...
I0920 15:31:15.081677       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 143 milliseconds
I0920 15:31:15.082780       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:31:15.083042       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-d94df931-324b-4db3-ac29-91a858763a58) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:31:15.096106       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=137932597
I0920 15:31:15.096349       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:31:15.096604       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:31:15.097069       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:31:15.097322       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.014128431 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:31:15.097431       1 cloudprovisioner.go:246] begin to create disk(pvc-d94df931-324b-4db3-ac29-91a858763a58) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone() maxShares(0)
I0920 15:31:15.115757       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:31:15.116093       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019206141 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:31:15.116209       1 cloudprovisioner.go:246] begin to create disk(pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone() maxShares(0)
I0920 15:31:15.121560       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023990151 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:15.121692       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-d94df931-324b-4db3-ac29-91a858763a58 StorageAccountType:StandardSSD_ZRS Size:10
I0920 15:31:15.123680       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:31:15.123780       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-2b876417-4866-4173-a2b6-9a5f2335716c)
... skipping 7 lines ...
I0920 15:31:15.127162       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:31:15.127250       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3)
I0920 15:31:15.144609       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028209961 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:15.144655       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b StorageAccountType:StandardSSD_ZRS Size:10
I0920 15:31:15.198553       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 202 milliseconds
I0920 15:31:15.209768       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2b876417-4866-4173-a2b6-9a5f2335716c/status 200 OK in 124 milliseconds
I0920 15:31:15.210628       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:15.211079       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:31:15.211117       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2)
I0920 15:31:15.215104       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=129762180
I0920 15:31:15.215424       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:31:15.215709       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-2b876417-4866-4173-a2b6-9a5f2335716c) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:31:15.219276       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:15.246115       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:15.246167       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:15.248616       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 44 milliseconds
I0920 15:31:15.260915       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:15.269715       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:15.276945       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64/status 200 OK in 39 milliseconds
I0920 15:31:15.277988       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3/status 200 OK in 23 milliseconds
I0920 15:31:15.280390       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=44364096
I0920 15:31:15.280793       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:31:15.281316       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:31:15.281181       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:31:15.282235       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.066163742 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:31:15.282511       1 cloudprovisioner.go:246] begin to create disk(pvc-2b876417-4866-4173-a2b6-9a5f2335716c) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone() maxShares(0)
I0920 15:31:15.284639       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=30026265
I0920 15:31:15.285202       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:31:15.285569       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:31:15.290604       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:15.297553       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:31:15.297997       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016225135 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:31:15.298389       1 cloudprovisioner.go:246] begin to create disk(pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone() maxShares(0)
I0920 15:31:15.300923       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:15.303481       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 33 milliseconds
I0920 15:31:15.304057       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:31:15.305175       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2/status 200 OK in 34 milliseconds
I0920 15:31:15.314767       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=44008594
I0920 15:31:15.317645       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:31:15.315781       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:31:15.317568       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:15.318479       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:31:15.323702       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.035515477 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:31:15.324394       1 cloudprovisioner.go:246] begin to create disk(pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone() maxShares(0)
I0920 15:31:15.333479       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.050675609 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:15.333751       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-2b876417-4866-4173-a2b6-9a5f2335716c StorageAccountType:StandardSSD_ZRS Size:10
I0920 15:31:15.340490       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:31:15.340902       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015649034 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:31:15.341142       1 cloudprovisioner.go:246] begin to create disk(pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone() maxShares(0)
I0920 15:31:15.352149       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.053479615 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:15.352615       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64 StorageAccountType:StandardSSD_ZRS Size:10
I0920 15:31:15.369877       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028450161 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:15.369935       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2 StorageAccountType:StandardSSD_ZRS Size:10
... skipping 9 lines ...
I0920 15:31:17.400581       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=2303983167
I0920 15:31:17.401207       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.2793489129999998 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:17.412492       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b/status 200 OK in 11 milliseconds
I0920 15:31:17.413678       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=13030528
I0920 15:31:17.413965       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=2317450396
I0920 15:31:17.414026       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=2455885094
I0920 15:31:17.416975       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:17.417045       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=2458568899
I0920 15:31:17.417218       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=2537823670
I0920 15:31:17.417277       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.538025571 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" result_code="succeeded"
I0920 15:31:17.417297       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-wftzv-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"}}
I0920 15:31:17.423888       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022352948 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:17.423925       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-d94df931-324b-4db3-ac29-91a858763a58 StorageAccountType:StandardSSD_ZRS Size:10
I0920 15:31:17.423984       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-d94df931-324b-4db3-ac29-91a858763a58) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-d94df931-324b-4db3-ac29-91a858763a58 kubernetes.io-created-for-pvc-name:test-volume-1-azuredisk-volume-tester-6rkjt-0 kubernetes.io-created-for-pvc-namespace:azuredisk-6803]) successfully" "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:31:17.424025       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=2341021846
I0920 15:31:17.441746       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-d94df931-324b-4db3-ac29-91a858763a58/status 200 OK in 16 milliseconds
I0920 15:31:17.443366       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:17.443868       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=2501952993
I0920 15:31:17.443925       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=2539132273
I0920 15:31:17.443980       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.539275673 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58" result_code="succeeded"
I0920 15:31:17.444001       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-d94df931-324b-4db3-ac29-91a858763a58","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-6rkjt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58"}}
I0920 15:31:17.444508       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=20412044
I0920 15:31:17.444801       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=2361899291
... skipping 2 lines ...
I0920 15:31:17.689617       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.336932038 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:17.697219       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.03742188 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:17.697260       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2 StorageAccountType:StandardSSD_ZRS Size:10
I0920 15:31:17.697315       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2 kubernetes.io-created-for-pvc-name:test-volume-3-azuredisk-volume-tester-wftzv-0 kubernetes.io-created-for-pvc-namespace:azuredisk-6803]) successfully" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:31:17.697356       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=2378928027
I0920 15:31:17.708898       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2/status 200 OK in 11 milliseconds
I0920 15:31:17.712108       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:17.712551       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022547149 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:17.713844       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64 StorageAccountType:StandardSSD_ZRS Size:10
I0920 15:31:17.713922       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64 kubernetes.io-created-for-pvc-name:test-volume-2-azuredisk-volume-tester-6rkjt-0 kubernetes.io-created-for-pvc-namespace:azuredisk-6803]) successfully" "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:31:17.713220       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=15795334
I0920 15:31:17.714308       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=2395934965
I0920 15:31:17.714495       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=2443747467
I0920 15:31:17.713284       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=2501369992
I0920 15:31:17.714751       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=2719212162
I0920 15:31:17.714949       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.719475262 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" result_code="succeeded"
I0920 15:31:17.714276       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=2432663543
I0920 15:31:17.714978       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-wftzv-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"}}
I0920 15:31:17.730202       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.353915674 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:17.733341       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64/status 200 OK in 17 milliseconds
I0920 15:31:17.734911       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:17.735106       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=2610480627
I0920 15:31:17.735172       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=2792513419
I0920 15:31:17.735343       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.792585719 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" result_code="succeeded"
I0920 15:31:17.735377       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-6rkjt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"}}
I0920 15:31:17.742995       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=27768960
I0920 15:31:17.743165       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=2461893507
... skipping 3 lines ...
I0920 15:31:17.754126       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3 kubernetes.io-created-for-pvc-name:test-volume-3-azuredisk-volume-tester-6rkjt-0 kubernetes.io-created-for-pvc-namespace:azuredisk-6803]) successfully" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:31:17.754339       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=2468656521
I0920 15:31:17.779448       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3/status 200 OK in 24 milliseconds
I0920 15:31:17.788553       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=34132174
I0920 15:31:17.788625       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=2503131695
I0920 15:31:17.788665       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=2534075162
I0920 15:31:17.789628       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:17.789722       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=2662296739
I0920 15:31:17.789898       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=2824328688
I0920 15:31:17.790236       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.824559988 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" result_code="succeeded"
I0920 15:31:17.790264       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-6rkjt-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"}}
I0920 15:31:17.823394       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.489451766 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:17.847843       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022660349 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:17.847882       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-2b876417-4866-4173-a2b6-9a5f2335716c StorageAccountType:StandardSSD_ZRS Size:10
I0920 15:31:17.847947       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-2b876417-4866-4173-a2b6-9a5f2335716c) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-2b876417-4866-4173-a2b6-9a5f2335716c kubernetes.io-created-for-pvc-name:test-volume-2-azuredisk-volume-tester-wftzv-0 kubernetes.io-created-for-pvc-namespace:azuredisk-6803]) successfully" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:31:17.847994       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=2632298574
I0920 15:31:17.863083       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2b876417-4866-4173-a2b6-9a5f2335716c/status 200 OK in 14 milliseconds
I0920 15:31:17.865557       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:17.865666       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=2741678210
I0920 15:31:17.865734       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=2928624912
I0920 15:31:17.865809       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.929345715 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c" result_code="succeeded"
I0920 15:31:17.865827       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2b876417-4866-4173-a2b6-9a5f2335716c","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-wftzv-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c"}}
I0920 15:31:17.869261       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=21202946
I0920 15:31:17.869580       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=2653891121
... skipping 14 lines ...
I0920 15:31:18.589319       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 26 milliseconds
I0920 15:31:18.590220       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3-1417k8s00000002-attachment)
I0920 15:31:18.600822       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-d94df931-324b-4db3-ac29-91a858763a58-1417k8s00000002-attachment/status 200 OK in 31 milliseconds
I0920 15:31:18.601816       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=32231769
I0920 15:31:18.602114       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:31:18.616084       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64-1417k8s00000002-attachment/status 200 OK in 41 milliseconds
I0920 15:31:18.616648       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:18.616688       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3-1417k8s00000002-attachment/status 200 OK in 26 milliseconds
I0920 15:31:18.616947       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=27941061
I0920 15:31:18.616987       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:31:18.617891       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=43671694
I0920 15:31:18.617929       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:31:18.622862       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:18.630733       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:18.641748       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.033551572 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:18.641819       1 azure_vmss_cache.go:332] Node 1417k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:31:18.652013       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.034000873 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:18.659875       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.042773492 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:18.660329       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01842694 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:18.660354       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
... skipping 18 lines ...
I0920 15:31:19.462152       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000000,disk.csi.azure.com/volume-name!=pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:31:19.463275       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 15:31:19.463584       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2b876417-4866-4173-a2b6-9a5f2335716c","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-wftzv-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c"}
I0920 15:31:19.464059       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000000,disk.csi.azure.com/volume-name!=pvc-2b876417-4866-4173-a2b6-9a5f2335716c." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:31:19.476329       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 13 milliseconds
I0920 15:31:19.479451       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b-1417k8s00000000-attachment)
I0920 15:31:19.481004       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:19.481927       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 15:31:19.482193       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-wftzv-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"}
I0920 15:31:19.482872       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000000,disk.csi.azure.com/volume-name!=pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:31:19.486379       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 21 milliseconds
I0920 15:31:19.505117       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b-1417k8s00000000-attachment/status 200 OK in 25 milliseconds
I0920 15:31:19.506964       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2b876417-4866-4173-a2b6-9a5f2335716c-1417k8s00000000-attachment)
I0920 15:31:19.507962       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:19.508210       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=29311163
I0920 15:31:19.508245       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:31:19.518352       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 34 milliseconds
I0920 15:31:19.519008       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2-1417k8s00000000-attachment)
I0920 15:31:19.526458       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-2b876417-4866-4173-a2b6-9a5f2335716c-1417k8s00000000-attachment/status 200 OK in 22 milliseconds
I0920 15:31:19.527581       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=23588050
I0920 15:31:19.527865       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:31:19.530323       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021953948 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:19.530588       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:31:19.533320       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:19.537401       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2-1417k8s00000000-attachment/status 200 OK in 17 milliseconds
I0920 15:31:19.540802       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:31:19.542172       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=22488948
I0920 15:31:19.542238       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:31:19.549772       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021546046 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:19.564005       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021594946 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:19.593804       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.062984135 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:19.593838       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
... skipping 13 lines ...
I0920 15:31:19.698134       1 azure_vmss_cache.go:332] Node 1417k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:31:19.717060       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01878744 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:19.717093       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:31:19.717145       1 azure_vmss_cache.go:332] Node 1417k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:31:19.730707       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-d94df931-324b-4db3-ac29-91a858763a58-1417k8s00000002-attachment/status 200 OK in 13 milliseconds
I0920 15:31:19.731571       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=14186330
I0920 15:31:19.732943       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:19.733188       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=1163952209
I0920 15:31:19.733385       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=1164464910
I0920 15:31:19.733525       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=1181242946
I0920 15:31:19.733732       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.181484647 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58" node="1417k8s00000002" result_code="succeeded"
I0920 15:31:19.733842       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 15:31:19.737558       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64-1417k8s00000002-attachment/status 200 OK in 19 milliseconds
I0920 15:31:19.739182       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3-1417k8s00000002-attachment/status 200 OK in 21 milliseconds
I0920 15:31:19.749310       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:19.749958       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=32023769
I0920 15:31:19.750283       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=32738371
I0920 15:31:19.750493       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=1160000400
I0920 15:31:19.750701       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=1160498202
I0920 15:31:19.750993       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=1188919763
I0920 15:31:19.751405       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.189463564 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" node="1417k8s00000002" result_code="succeeded"
I0920 15:31:19.751330       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.034142274 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:19.751933       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:31:19.752098       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3:AttachDiskOptions{diskName: "pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58:AttachDiskOptions{diskName: "pvc-d94df931-324b-4db3-ac29-91a858763a58", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64:AttachDiskOptions{diskName: "pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64", lun: 1}]
I0920 15:31:19.752402       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3:AttachDiskOptions{diskName: "pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58:AttachDiskOptions{diskName: "pvc-d94df931-324b-4db3-ac29-91a858763a58", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64:AttachDiskOptions{diskName: "pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64", lun: 1}])
I0920 15:31:19.751623       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0920 15:31:19.759644       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:19.760022       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=1183740752
I0920 15:31:19.760390       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=1184447553
I0920 15:31:19.760829       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=1203706394
I0920 15:31:19.761163       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.204074595 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" node="1417k8s00000002" result_code="succeeded"
I0920 15:31:19.761485       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0920 15:31:20.552633       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.799847124 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 15:31:20.552674       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3:AttachDiskOptions{diskName: "pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58:AttachDiskOptions{diskName: "pvc-d94df931-324b-4db3-ac29-91a858763a58", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64:AttachDiskOptions{diskName: "pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 15:31:20.594918       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-y9dto3pe|1417k8s00000000" delay="1s"
I0920 15:31:20.595239       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:31:20.631498       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.036198678 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:31:20.631529       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:31:20.631570       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:31:20.643926       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-2b876417-4866-4173-a2b6-9a5f2335716c-1417k8s00000000-attachment/status 200 OK in 11 milliseconds
I0920 15:31:20.647440       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:20.647810       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=1140696659
I0920 15:31:20.648150       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=1141177260
I0920 15:31:20.648469       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=1184471254
I0920 15:31:20.648862       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.184799454 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c" node="1417k8s00000000" result_code="succeeded"
I0920 15:31:20.649061       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0920 15:31:20.650286       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018675941 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 2 lines ...
I0920 15:31:20.651451       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c:AttachDiskOptions{diskName: "pvc-2b876417-4866-4173-a2b6-9a5f2335716c", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2:AttachDiskOptions{diskName: "pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b:AttachDiskOptions{diskName: "pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b", lun: 0}]
I0920 15:31:20.652056       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c:AttachDiskOptions{diskName: "pvc-2b876417-4866-4173-a2b6-9a5f2335716c", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2:AttachDiskOptions{diskName: "pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b:AttachDiskOptions{diskName: "pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b", lun: 0}])
I0920 15:31:20.654590       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b-1417k8s00000000-attachment/status 200 OK in 22 milliseconds
I0920 15:31:20.656608       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2-1417k8s00000000-attachment/status 200 OK in 23 milliseconds
I0920 15:31:20.657176       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=24491953
I0920 15:31:20.660442       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=28486561
I0920 15:31:20.665745       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:20.667010       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=1186009056
I0920 15:31:20.667398       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=1187840460
I0920 15:31:20.667677       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=1205780399
I0920 15:31:20.668080       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.2061318 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" node="1417k8s00000000" result_code="succeeded"
I0920 15:31:20.668420       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 15:31:20.669506       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:20.670689       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=1150503680
I0920 15:31:20.670801       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=1151760483
I0920 15:31:20.670915       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=1188171161
I0920 15:31:20.671029       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.188284961 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" node="1417k8s00000000" result_code="succeeded"
I0920 15:31:20.671111       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"2"}}
I0920 15:31:20.673051       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 15:31:20.673079       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000000","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-2b876417-4866-4173-a2b6-9a5f2335716c","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-wftzv-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c"}
I0920 15:31:20.673697       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000000,disk.csi.azure.com/volume-name!=pvc-2b876417-4866-4173-a2b6-9a5f2335716c." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="46f399a0-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:31:20.673991       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="46f399a0-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=62100
I0920 15:31:20.674094       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="46f399a0-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=494201
I0920 15:31:20.674194       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.000751102 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c" node="1417k8s00000000" result_code="succeeded"
I0920 15:31:20.674222       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0920 15:31:20.864868       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.212579358 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 15:31:20.864907       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c:AttachDiskOptions{diskName: "pvc-2b876417-4866-4173-a2b6-9a5f2335716c", lun: 1} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2:AttachDiskOptions{diskName: "pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2", lun: 2} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b:AttachDiskOptions{diskName: "pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 15:31:21.331656       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 15:31:21.336338       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:31:21.336966       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:31:23.345197       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 15:31:23.350020       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:31:23.350556       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 850 lines ...
I0920 15:36:04.386040       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2-1417k8s00000000-attachment) detachment" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:36:04.387797       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-2b876417-4866-4173-a2b6-9a5f2335716c-1417k8s00000000-attachment/status 200 OK in 18 milliseconds
I0920 15:36:04.388365       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=19466662
I0920 15:36:04.388763       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:36:04.389055       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c from node 1417k8s00000000" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:36:04.389252       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:36:04.392789       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:04.405473       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2-1417k8s00000000-attachment/status 200 OK in 18 milliseconds
I0920 15:36:04.406841       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=20434065
I0920 15:36:04.419930       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2-1417k8s00000000-attachment 200 OK in 12 milliseconds
I0920 15:36:04.420829       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2) scheduled in 5m0s." "disk.csi.azure.com/request-id"="f013fedf-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:36:04.421546       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2-1417k8s00000000-attachment)
I0920 15:36:04.423394       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0920 15:36:04.423551       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000000","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"}
I0920 15:36:04.424105       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b-1417k8s00000000-attachment) detachment" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:36:04.435421       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2-1417k8s00000000-attachment/status 200 OK in 14 milliseconds
I0920 15:36:04.439102       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:04.439907       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=19545862
I0920 15:36:04.440737       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:36:04.441071       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2 from node 1417k8s00000000" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:36:04.442319       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b-1417k8s00000000-attachment/status 200 OK in 17 milliseconds
I0920 15:36:04.444044       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=19682863
I0920 15:36:04.451934       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b) scheduled in 5m0s." "disk.csi.azure.com/request-id"="f018bb13-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
... skipping 4 lines ...
I0920 15:36:04.456908       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b-1417k8s00000000-attachment 200 OK in 12 milliseconds
I0920 15:36:04.457788       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b-1417k8s00000000-attachment)
I0920 15:36:04.461593       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b-1417k8s00000000-attachment/status 200 OK in 9 milliseconds
I0920 15:36:04.462768       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=11177935
I0920 15:36:04.462815       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:36:04.463150       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b from node 1417k8s00000000" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:36:04.465720       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:04.538382       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.083687667 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:04.538419       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:36:04.538464       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:36:04.556865       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018360359 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:04.556895       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:36:04.596661       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.143165757 request="vmssvm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 51 lines ...
I0920 15:36:25.986748       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=21597733315
I0920 15:36:25.987023       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=21523636679
I0920 15:36:25.987217       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2) succeeded
I0920 15:36:25.987388       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2 from node 1417k8s00000000 successfully" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:36:25.987428       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=21546438551
I0920 15:36:26.001804       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-2b876417-4866-4173-a2b6-9a5f2335716c-1417k8s00000000-attachment 200 OK in 14 milliseconds
I0920 15:36:26.003730       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:36:26.004076       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=21631387115
I0920 15:36:26.004460       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=21634353825
I0920 15:36:26.004799       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="f0054093-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=21680541971
I0920 15:36:26.005154       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=21.681106073 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c" node="1417k8s00000000" result_code="succeeded"
I0920 15:36:26.005370       1 utils.go:85] GRPC response: {}
I0920 15:36:26.011288       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463b0768-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=24461967
... skipping 4 lines ...
I0920 15:36:26.013835       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=21572583723
I0920 15:36:26.014116       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=21593739990
I0920 15:36:26.014448       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b-1417k8s00000000-attachment 200 OK in 26 milliseconds
I0920 15:36:26.015034       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=27717076
I0920 15:36:26.015302       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=21552382557
I0920 15:36:26.015599       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=21563955994
I0920 15:36:26.025299       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:36:26.025637       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=21603690917
I0920 15:36:26.025993       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463de56a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=21605106721
I0920 15:36:26.026326       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="f00eaf60-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=21640239234
I0920 15:36:26.026620       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=21.640634034 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" node="1417k8s00000000" result_code="succeeded"
I0920 15:36:26.026820       1 utils.go:85] GRPC response: {}
I0920 15:36:26.027690       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:36:26.028943       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=21571060511
I0920 15:36:26.029198       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="463ab53f-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=21571402413
I0920 15:36:26.029460       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="f0147c06-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=21605409620
I0920 15:36:26.029761       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=21.605918522 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" node="1417k8s00000000" result_code="succeeded"
I0920 15:36:26.030025       1 utils.go:85] GRPC response: {}
I0920 15:36:26.037580       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 26 lines ...
I0920 15:36:29.278193       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b/status 200 OK in 10 milliseconds
I0920 15:36:29.282508       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="fee33c10-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=15411242
I0920 15:36:29.282535       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b)
I0920 15:36:29.295909       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0920 15:36:29.295992       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c"}
I0920 15:36:29.296114       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c)
I0920 15:36:29.299377       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:29.299570       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b 200 OK in 15 milliseconds
I0920 15:36:29.315790       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b/status 200 OK in 14 milliseconds
I0920 15:36:29.318795       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2b876417-4866-4173-a2b6-9a5f2335716c/status 200 OK in 21 milliseconds
I0920 15:36:29.330129       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0920 15:36:29.330153       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"}
I0920 15:36:29.330241       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2)
... skipping 3 lines ...
I0920 15:36:29.348788       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b)" "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:36:29.348861       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b)" "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:36:29.348961       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b"
I0920 15:36:29.349066       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=372901
I0920 15:36:29.351507       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="fee7b0ae-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=55185552
I0920 15:36:29.351729       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-2b876417-4866-4173-a2b6-9a5f2335716c)
I0920 15:36:29.362233       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:29.362459       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:29.364651       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2/status 200 OK in 33 milliseconds
I0920 15:36:29.365522       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="feece463-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=35117996
I0920 15:36:29.365793       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2)
I0920 15:36:29.366244       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 16 milliseconds
I0920 15:36:29.368040       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:36:29.380377       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.031194686 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:29.385195       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2b876417-4866-4173-a2b6-9a5f2335716c 200 OK in 33 milliseconds
I0920 15:36:29.386662       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:29.394894       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2 200 OK in 28 milliseconds
I0920 15:36:29.396948       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:29.405719       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:29.406264       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2b876417-4866-4173-a2b6-9a5f2335716c/status 200 OK in 19 milliseconds
I0920 15:36:29.410903       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=24452467
I0920 15:36:29.411125       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:36:29.411453       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-2b876417-4866-4173-a2b6-9a5f2335716c)" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:36:29.411716       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-2b876417-4866-4173-a2b6-9a5f2335716c)" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
I0920 15:36:29.411843       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-2b876417-4866-4173-a2b6-9a5f2335716c." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c"
... skipping 2 lines ...
I0920 15:36:29.416381       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=17056747
I0920 15:36:29.416423       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:36:29.416500       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2)" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:36:29.416528       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2)" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:36:29.416569       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"
I0920 15:36:29.416621       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=121801
I0920 15:36:29.418420       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:29.445104       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.033118491 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:29.494950       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.078242915 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:31.380410       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 10 milliseconds
I0920 15:36:31.387057       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:36:31.388096       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:36:31.842856       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:36:31.853515       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:36:31.856715       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:36:31.856968       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:31.857043       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:31.857073       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:32.389196       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:36:32.389322       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:36:32.390297       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:36:33.407634       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 18 milliseconds
I0920 15:36:33.413696       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:36:33.414329       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:36:34.522391       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.141844586 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:34.522422       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b
I0920 15:36:34.522552       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=5173296572
I0920 15:36:34.545236       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:36:34.545514       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="fee33c10-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=5245602395
I0920 15:36:34.545607       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="fee33c10-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=5278499885
I0920 15:36:34.545670       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b) returned with <nil>
I0920 15:36:34.545875       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.278848086 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" result_code="succeeded"
I0920 15:36:34.545936       1 utils.go:85] GRPC response: {}
I0920 15:36:34.549129       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b 200 OK in 26 milliseconds
I0920 15:36:34.550394       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=27705105
I0920 15:36:34.550684       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=5202008280
I0920 15:36:34.550893       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "disk.csi.azure.com/request-id"="437f7668-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=5249815712
I0920 15:36:34.630686       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.185521319 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:34.630717       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c
I0920 15:36:34.630777       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=5218783010
I0920 15:36:34.643913       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:36:34.644889       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="fee7b0ae-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=5256832529
I0920 15:36:34.645064       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="fee7b0ae-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=5348727383
I0920 15:36:34.645092       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c) returned with <nil>
I0920 15:36:34.645278       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.348995684 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-2b876417-4866-4173-a2b6-9a5f2335716c" result_code="succeeded"
I0920 15:36:34.645311       1 utils.go:85] GRPC response: {}
I0920 15:36:34.647045       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-2b876417-4866-4173-a2b6-9a5f2335716c 200 OK in 15 milliseconds
I0920 15:36:34.648473       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=17632367
I0920 15:36:34.648628       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=5237185180
I0920 15:36:34.648776       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "disk.csi.azure.com/request-id"="438848fa-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=5262309549
I0920 15:36:34.653230       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.158220968 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:34.653472       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2
I0920 15:36:34.653809       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=5236995885
I0920 15:36:34.706461       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2 200 OK in 51 milliseconds
I0920 15:36:34.710653       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:36:34.711073       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="feece463-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=5312110451
I0920 15:36:34.711340       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="feece463-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=5380935440
I0920 15:36:34.711518       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=57446118
I0920 15:36:34.711590       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=5295106106
I0920 15:36:34.711641       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "disk.csi.azure.com/request-id"="43912d08-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=5312333453
I0920 15:36:34.711519       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2) returned with <nil>
... skipping 66 lines ...
I0920 15:36:58.590872       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000002","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"}
I0920 15:36:58.591362       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64-1417k8s00000002-attachment) detachment" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:36:58.599309       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3) scheduled in 5m0s." "disk.csi.azure.com/request-id"="105ef72b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:36:58.600246       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3-1417k8s00000002-attachment)
I0920 15:36:58.620481       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-d94df931-324b-4db3-ac29-91a858763a58-1417k8s00000002-attachment 200 OK in 30 milliseconds
I0920 15:36:58.620619       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-d94df931-324b-4db3-ac29-91a858763a58-1417k8s00000002-attachment)
I0920 15:36:58.623421       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:58.624066       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-d94df931-324b-4db3-ac29-91a858763a58) scheduled in 5m0s." "disk.csi.azure.com/request-id"="1062c0bd-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:36:58.633119       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64-1417k8s00000002-attachment/status 200 OK in 41 milliseconds
I0920 15:36:58.633455       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=41829048
I0920 15:36:58.635332       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:58.635360       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3-1417k8s00000002-attachment/status 200 OK in 36 milliseconds
I0920 15:36:58.635650       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=37141320
I0920 15:36:58.635686       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:36:58.635787       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3 from node 1417k8s00000002" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:36:58.635805       1 azure_vmss_cache.go:332] Node 1417k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:36:58.644367       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-d94df931-324b-4db3-ac29-91a858763a58-1417k8s00000002-attachment/status 200 OK in 20 milliseconds
I0920 15:36:58.646489       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64-1417k8s00000002-attachment 200 OK in 12 milliseconds
I0920 15:36:58.647143       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=23601440
I0920 15:36:58.647186       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:36:58.647345       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58 from node 1417k8s00000002" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:36:58.647663       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64-1417k8s00000002-attachment)
I0920 15:36:58.650324       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64) scheduled in 5m0s." "disk.csi.azure.com/request-id"="1066c29f-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:36:58.650846       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:58.654803       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:58.662733       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64-1417k8s00000002-attachment/status 200 OK in 12 milliseconds
I0920 15:36:58.663346       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=13556980
I0920 15:36:58.663401       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:36:58.663495       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64 from node 1417k8s00000002" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:36:58.664818       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:36:58.669100       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.032689994 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:58.669334       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:36:58.669590       1 azure_vmss_cache.go:332] Node 1417k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:36:58.669601       1 azure_vmss_cache.go:332] Node 1417k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:36:58.687328       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017600505 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:36:58.687420       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
... skipping 13 lines ...
I0920 15:36:59.721063       1 azure_controller_vmss.go:197] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000002) - detach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3:pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3 /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58:pvc-d94df931-324b-4db3-ac29-91a858763a58 /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64:pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64])
I0920 15:37:01.660078       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 15:37:01.671179       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 10 milliseconds
I0920 15:37:01.671610       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:37:01.843344       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:37:01.854509       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:37:01.854628       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:01.854672       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:01.854697       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:01.857677       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:37:02.390075       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:37:02.390076       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:37:02.390450       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:37:03.682494       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 10 milliseconds
I0920 15:37:03.688341       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
... skipping 26 lines ...
I0920 15:37:15.881532       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3 from node 1417k8s00000002 successfully" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:37:15.881613       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=17245855945
I0920 15:37:15.882033       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-d94df931-324b-4db3-ac29-91a858763a58, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58) succeeded
I0920 15:37:15.882116       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58 from node 1417k8s00000002 successfully" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:37:15.882231       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=17234978278
I0920 15:37:15.910301       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3-1417k8s00000002-attachment 200 OK in 28 milliseconds
I0920 15:37:15.913397       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:15.913524       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=17312914136
I0920 15:37:15.913654       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=17313509239
I0920 15:37:15.913741       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="105552dd-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=17377664419
I0920 15:37:15.913821       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.377958521 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" node="1417k8s00000002" result_code="succeeded"
I0920 15:37:15.913876       1 utils.go:85] GRPC response: {}
I0920 15:37:15.914716       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=32980985
I0920 15:37:15.914826       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=17279043130
I0920 15:37:15.914929       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b169ba-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=17316384752
I0920 15:37:15.920239       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-d94df931-324b-4db3-ac29-91a858763a58-1417k8s00000002-attachment 200 OK in 37 milliseconds
I0920 15:37:15.924818       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=42453510
I0920 15:37:15.924915       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=17277642888
I0920 15:37:15.924999       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=17301436029
I0920 15:37:15.925664       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:15.925786       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=17305043048
I0920 15:37:15.925903       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45afe82a-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=17305270249
I0920 15:37:15.926015       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="105a2ad1-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=17358169563
I0920 15:37:15.926126       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.358445664 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58" node="1417k8s00000002" result_code="succeeded"
I0920 15:37:15.926154       1 utils.go:85] GRPC response: {}
I0920 15:37:15.930738       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:15.930934       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=17283162401
I0920 15:37:15.931045       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="45b0a5e9-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=17283365202
I0920 15:37:15.931134       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="105dc317-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=17339753336
I0920 15:37:15.931213       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=17.339902236 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" node="1417k8s00000002" result_code="succeeded"
I0920 15:37:15.931233       1 utils.go:85] GRPC response: {}
I0920 15:37:15.935069       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64-1417k8s00000002-attachment 200 OK in 53 milliseconds
... skipping 19 lines ...
I0920 15:37:20.386125       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="1d58b30b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=17557645
I0920 15:37:20.386150       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-d94df931-324b-4db3-ac29-91a858763a58)
I0920 15:37:20.389386       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0920 15:37:20.389425       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"}
I0920 15:37:20.389684       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64)
I0920 15:37:20.398004       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-d94df931-324b-4db3-ac29-91a858763a58 200 OK in 11 milliseconds
I0920 15:37:20.400441       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:20.412013       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64/status 200 OK in 20 milliseconds
I0920 15:37:20.412559       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="1d5bffb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=22355158
I0920 15:37:20.412586       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64)
I0920 15:37:20.413026       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0920 15:37:20.413057       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"}
I0920 15:37:20.413208       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3)
I0920 15:37:20.440479       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64 200 OK in 27 milliseconds
I0920 15:37:20.445004       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3/status 200 OK in 30 milliseconds
I0920 15:37:20.448856       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-d94df931-324b-4db3-ac29-91a858763a58/status 200 OK in 46 milliseconds
I0920 15:37:20.453010       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:20.455299       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="1d5f8bc6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=41835808
I0920 15:37:20.455336       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3)
I0920 15:37:20.457915       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=55996345
I0920 15:37:20.457967       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:37:20.460210       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:20.460253       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:20.460371       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-d94df931-324b-4db3-ac29-91a858763a58)" "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:37:20.460406       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-d94df931-324b-4db3-ac29-91a858763a58)" "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:37:20.460542       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-d94df931-324b-4db3-ac29-91a858763a58." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58"
I0920 15:37:20.460640       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=2562407
I0920 15:37:20.467546       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3 200 OK in 12 milliseconds
I0920 15:37:20.483011       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022254157 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:20.494834       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:20.524457       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64/status 200 OK in 72 milliseconds
I0920 15:37:20.526107       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:20.526632       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=75186394
I0920 15:37:20.526693       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:37:20.526908       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64)" "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:37:20.526956       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64)" "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:37:20.527052       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64"
I0920 15:37:20.527186       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=347301
I0920 15:37:20.589517       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3/status 200 OK in 87 milliseconds
I0920 15:37:20.591203       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:20.591982       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=90665934
I0920 15:37:20.592258       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:37:20.592808       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3)" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:37:20.593008       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3)" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:37:20.593316       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3"
I0920 15:37:20.593611       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=919002
... skipping 9 lines ...
I0920 15:37:25.676190       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58
I0920 15:37:25.676251       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=5215491064
I0920 15:37:25.697677       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-d94df931-324b-4db3-ac29-91a858763a58 200 OK in 20 milliseconds
I0920 15:37:25.701752       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=25367639
I0920 15:37:25.701811       1 azvolume.go:259]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=5243790110
I0920 15:37:25.701856       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="43835b92-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=5299955255
I0920 15:37:25.702450       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:25.703069       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="1d58b30b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=5299904553
I0920 15:37:25.703136       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "disk.csi.azure.com/request-id"="1d58b30b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=5334597043
I0920 15:37:25.703152       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58) returned with <nil>
I0920 15:37:25.703199       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.334774843 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58" result_code="succeeded"
I0920 15:37:25.703224       1 utils.go:85] GRPC response: {}
I0920 15:37:25.778140       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.136352354 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:25.778176       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3
I0920 15:37:25.778253       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=5184301377
I0920 15:37:25.791389       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3 200 OK in 12 milliseconds
I0920 15:37:25.795881       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:25.795957       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="1d5f8bc6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=5327570827
I0920 15:37:25.796005       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="1d5f8bc6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=5382593969
I0920 15:37:25.796027       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3) returned with <nil>
I0920 15:37:25.796067       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.3828347690000005 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" result_code="succeeded"
I0920 15:37:25.796097       1 utils.go:85] GRPC response: {}
I0920 15:37:25.796725       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "disk.csi.azure.com/request-id"="438ca140-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=18421228
... skipping 2 lines ...
I0920 15:37:25.836491       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:37:25.847213       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0920 15:37:25.847711       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:37:25.870122       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.271188405 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:25.870156       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64
I0920 15:37:25.870212       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="43892251-38f9-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=5342909890
I0920 15:37:25.884105       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:25.884520       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="1d5bffb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=5443735435
I0920 15:37:25.884805       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "disk.csi.azure.com/request-id"="1d5bffb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=5494625465
I0920 15:37:25.885302       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64) returned with <nil>
I0920 15:37:25.885605       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.495861568 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" result_code="succeeded"
I0920 15:37:25.885842       1 utils.go:85] GRPC response: {}
I0920 15:37:25.891521       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64 200 OK in 20 milliseconds
... skipping 41 lines ...
I0920 15:37:35.314555       1 shared_state.go:419]  "msg"="Pod spec of pod azuredisk-volume-tester-dm6wd-0 is: {Volumes:[{Name:test-volume-2 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-2-azuredisk-volume-tester-dm6wd-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-1-azuredisk-volume-tester-dm6wd-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-p8pl9 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}] InitContainers:[] Containers:[{Name:volume-tester Image:mcr.microsoft.com/windows/servercore:ltsc2019 Command:[powershell.exe] Args:[-Command while (1) { Add-Content -Encoding Ascii C:\\mnt\\test-1\\data.txt $(Get-Date -Format u); sleep 3600 }] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:test-volume-1 ReadOnly:false MountPath:/mnt/test-1 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:test-volume-2 ReadOnly:false MountPath:/mnt/test-2 SubPath: MountPropagation:<nil> SubPathExpr:} {Name:kube-api-access-p8pl9 ReadOnly:true MountPath:/var/run/secrets/kubernetes.io/serviceaccount SubPath: MountPropagation:<nil> SubPathExpr:}] VolumeDevices:[] LivenessProbe:nil ReadinessProbe:nil StartupProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}] EphemeralContainers:[] RestartPolicy:Always TerminationGracePeriodSeconds:0xc001054840 ActiveDeadlineSeconds:<nil> DNSPolicy:ClusterFirst NodeSelector:map[kubernetes.io/os:windows] ServiceAccountName:default DeprecatedServiceAccount:default AutomountServiceAccountToken:<nil> NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:nil,} ImagePullSecrets:[] Hostname:azuredisk-volume-tester-dm6wd-0 Subdomain: Affinity:nil SchedulerName:csi-azuredisk-scheduler-extender Tolerations:[{Key:node.kubernetes.io/not-ready Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001054848} {Key:node.kubernetes.io/unreachable Operator:Exists Value: Effect:NoExecute TolerationSeconds:0xc001054850}] HostAliases:[] PriorityClassName: Priority:0xc001054858 DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:0xc00105485c PreemptionPolicy:0xc0002ce0a0 Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil}. With volumes: [{Name:test-volume-2 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-2-azuredisk-volume-tester-dm6wd-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:test-volume-1 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:&PersistentVolumeClaimVolumeSource{ClaimName:test-volume-1-azuredisk-volume-tester-dm6wd-0,ReadOnly:false,} RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}} {Name:kube-api-access-p8pl9 VolumeSource:{HostPath:nil EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:&ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" 
I0920 15:37:35.314751       1 shared_state.go:457]  "msg"="Pod azuredisk-volume-tester-dm6wd-0: Skipping Volume {kube-api-access-p8pl9 {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil nil}}. No persistent volume exists." 
I0920 15:37:35.314780       1 shared_state.go:483]  "msg"="Storing pod azuredisk-volume-tester-dm6wd-0 and claim [] to podToClaimsMap map." 
I0920 15:37:35.330113       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 58 milliseconds
I0920 15:37:35.330436       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "disk.csi.azure.com/request-id"="263abeb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e"
I0920 15:37:35.330457       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-83cdeaff-ab63-4cc8-869b-94403969e88e)
I0920 15:37:35.335241       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:35.339371       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 40 milliseconds
I0920 15:37:35.345001       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb/status 200 OK in 54 milliseconds
I0920 15:37:35.345787       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "disk.csi.azure.com/request-id"="263eda86-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5"
I0920 15:37:35.345876       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5)
I0920 15:37:35.352673       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 15:37:35.352836       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":274877906944},"name":"pvc-61797b4c-2183-455b-be8b-79b20c92eace","parameters":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-61797b4c-2183-455b-be8b-79b20c92eace","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-dm6wd-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3778","maxShares":"2","skuname":"Premium_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
... skipping 3 lines ...
I0920 15:37:35.372054       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "disk.csi.azure.com/request-id"="263abeb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e"
I0920 15:37:35.372158       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-83cdeaff-ab63-4cc8-869b-94403969e88e) is ZRS(Premium_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "disk.csi.azure.com/request-id"="263abeb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e"
I0920 15:37:35.379966       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi.azure.com/request-id"="2638f880-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=91751041
I0920 15:37:35.380010       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi.azure.com/request-id"="2638f880-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb"
I0920 15:37:35.380111       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb) is ZRS(Premium_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi.azure.com/request-id"="2638f880-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb"
I0920 15:37:35.384432       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 30 milliseconds
I0920 15:37:35.387302       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:37:35.387400       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01514154 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:37:35.387427       1 cloudprovisioner.go:246] begin to create disk(pvc-83cdeaff-ab63-4cc8-869b-94403969e88e) account type(Premium_ZRS) rg(kubetest-y9dto3pe) location() size(256) selectedAvailabilityZone() maxShares(2)
I0920 15:37:35.390996       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi.azure.com/request-id"="26473646-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace"
I0920 15:37:35.391023       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-61797b4c-2183-455b-be8b-79b20c92eace)
I0920 15:37:35.392148       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:37:35.392269       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.012040431 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:37:35.392338       1 cloudprovisioner.go:246] begin to create disk(pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb) account type(Premium_ZRS) rg(kubetest-y9dto3pe) location() size(256) selectedAvailabilityZone() maxShares(2)
I0920 15:37:35.415582       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028102474 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:35.415625       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-83cdeaff-ab63-4cc8-869b-94403969e88e StorageAccountType:Premium_ZRS Size:256
I0920 15:37:35.419314       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.026895071 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:35.419360       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb StorageAccountType:Premium_ZRS Size:256
I0920 15:37:35.422285       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:35.429110       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:35.433722       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:35.434052       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:35.464824       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5/status 200 OK in 40 milliseconds
I0920 15:37:35.487306       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-61797b4c-2183-455b-be8b-79b20c92eace/status 200 OK in 55 milliseconds
I0920 15:37:35.496752       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "disk.csi.azure.com/request-id"="263eda86-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "latency"=72913692
I0920 15:37:35.496801       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "disk.csi.azure.com/request-id"="263eda86-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5"
I0920 15:37:35.497063       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5) is ZRS(Premium_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "disk.csi.azure.com/request-id"="263eda86-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5"
I0920 15:37:35.498424       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:35.499834       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi.azure.com/request-id"="26473646-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=68131779
I0920 15:37:35.499871       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi.azure.com/request-id"="26473646-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace"
I0920 15:37:35.499953       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-61797b4c-2183-455b-be8b-79b20c92eace) is ZRS(Premium_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi.azure.com/request-id"="26473646-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace"
I0920 15:37:35.501404       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:35.515017       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:37:35.516528       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019350151 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:37:35.516979       1 cloudprovisioner.go:246] begin to create disk(pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5) account type(Premium_ZRS) rg(kubetest-y9dto3pe) location() size(256) selectedAvailabilityZone() maxShares(2)
I0920 15:37:35.530203       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-61797b4c-2183-455b-be8b-79b20c92eace, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-61797b4c-2183-455b-be8b-79b20c92eace' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 15:37:35.530282       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.03025208 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 15:37:35.530310       1 cloudprovisioner.go:246] begin to create disk(pvc-61797b4c-2183-455b-be8b-79b20c92eace) account type(Premium_ZRS) rg(kubetest-y9dto3pe) location() size(256) selectedAvailabilityZone() maxShares(2)
I0920 15:37:35.558961       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.04175691 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:35.559001       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5 StorageAccountType:Premium_ZRS Size:256
I0920 15:37:35.568394       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.038029 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:35.568476       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-61797b4c-2183-455b-be8b-79b20c92eace StorageAccountType:Premium_ZRS Size:256
... skipping 2 lines ...
I0920 15:37:35.919939       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:37:37.616479       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.200794796 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:37.661856       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.045313319 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:37.661903       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-83cdeaff-ab63-4cc8-869b-94403969e88e StorageAccountType:Premium_ZRS Size:256
I0920 15:37:37.662011       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-83cdeaff-ab63-4cc8-869b-94403969e88e) account type(Premium_ZRS) rg(kubetest-y9dto3pe) location() size(256) tags(map[kubernetes.io-created-for-pv-name:pvc-83cdeaff-ab63-4cc8-869b-94403969e88e kubernetes.io-created-for-pvc-name:test-volume-1-azuredisk-volume-tester-ccwqb-0 kubernetes.io-created-for-pvc-namespace:azuredisk-3778]) successfully" "csi.storage.k8s.io/pv/name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "disk.csi.azure.com/request-id"="263abeb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e"
I0920 15:37:37.662141       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "disk.csi.azure.com/request-id"="263abeb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=2289990931
I0920 15:37:37.675830       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:37.676035       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "disk.csi.azure.com/request-id"="263abeb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=2345439577
I0920 15:37:37.676115       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "disk.csi.azure.com/request-id"="263abeb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=2404358132
I0920 15:37:37.676263       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.404476833 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" result_code="succeeded"
I0920 15:37:37.676331       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":274877906944,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-83cdeaff-ab63-4cc8-869b-94403969e88e","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-ccwqb-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3778","maxShares":"2","requestedsizegib":"256","skuname":"Premium_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e"}}
I0920 15:37:37.678596       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e/status 200 OK in 16 milliseconds
I0920 15:37:37.679865       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "disk.csi.azure.com/request-id"="263abeb6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=17652246
... skipping 23 lines ...
I0920 15:37:37.902910       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi.azure.com/request-id"="26473646-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=46521523
I0920 15:37:37.903206       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi.azure.com/request-id"="26473646-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=2403248130
I0920 15:37:37.903458       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi.azure.com/request-id"="26473646-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=2471766210
I0920 15:37:37.904628       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi.azure.com/request-id"="2638f880-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=44993718
I0920 15:37:37.904904       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi.azure.com/request-id"="2638f880-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=2524822149
I0920 15:37:37.905165       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi.azure.com/request-id"="2638f880-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=2616972092
I0920 15:37:37.913378       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:37.913700       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi.azure.com/request-id"="26473646-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=2522555343
I0920 15:37:37.913954       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi.azure.com/request-id"="26473646-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=2560523044
I0920 15:37:37.914208       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.560992245 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-61797b4c-2183-455b-be8b-79b20c92eace" result_code="succeeded"
I0920 15:37:37.914451       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":274877906944,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-61797b4c-2183-455b-be8b-79b20c92eace","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-dm6wd-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3778","maxShares":"2","requestedsizegib":"256","skuname":"Premium_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-61797b4c-2183-455b-be8b-79b20c92eace"}}
I0920 15:37:37.922608       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:37.922933       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi.azure.com/request-id"="2638f880-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=2633335635
I0920 15:37:37.923260       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi.azure.com/request-id"="2638f880-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=2663101013
I0920 15:37:37.923536       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.663438014 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" result_code="succeeded"
I0920 15:37:37.923795       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":274877906944,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb","csi.storage.k8s.io/pvc/name":"test-volume-1-azuredisk-volume-tester-dm6wd-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3778","maxShares":"2","requestedsizegib":"256","skuname":"Premium_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb"}}
I0920 15:37:37.929286       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:37.929532       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "disk.csi.azure.com/request-id"="263eda86-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "latency"=2583475004
I0920 15:37:37.929860       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "disk.csi.azure.com/request-id"="263eda86-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "latency"=2631156330
I0920 15:37:37.930105       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.63148223 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" result_code="succeeded"
I0920 15:37:37.930350       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":274877906944,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5","csi.storage.k8s.io/pvc/name":"test-volume-2-azuredisk-volume-tester-ccwqb-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-3778","maxShares":"2","requestedsizegib":"256","skuname":"Premium_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5"}}
I0920 15:37:37.966424       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 45 milliseconds
I0920 15:37:38.002887       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 26 milliseconds
... skipping 19 lines ...
I0920 15:37:38.845016       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb-1417k8s00000003-attachment)
I0920 15:37:38.847369       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 13 milliseconds
I0920 15:37:38.848723       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 27 milliseconds
I0920 15:37:38.849258       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5-1417k8s00000001-attachment)
I0920 15:37:38.850383       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-61797b4c-2183-455b-be8b-79b20c92eace-1417k8s00000003-attachment)
I0920 15:37:38.852200       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-83cdeaff-ab63-4cc8-869b-94403969e88e-1417k8s00000001-attachment)
I0920 15:37:38.856761       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:38.878613       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:38.878782       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:38.878811       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:38.889885       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:38.889994       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-61797b4c-2183-455b-be8b-79b20c92eace-1417k8s00000003-attachment/status 200 OK in 34 milliseconds
I0920 15:37:38.890528       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="2857f07b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=35413396
I0920 15:37:38.890690       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="2857f07b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace"
I0920 15:37:38.892257       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e-1417k8s00000001-attachment/status 200 OK in 26 milliseconds
I0920 15:37:38.894180       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb-1417k8s00000003-attachment/status 200 OK in 30 milliseconds
I0920 15:37:38.895839       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="2857fd7d-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=31108884
I0920 15:37:38.896150       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="2857fd7d-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e"
I0920 15:37:38.895904       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="28590ce8-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=36395798
I0920 15:37:38.896493       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="28590ce8-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb"
I0920 15:37:38.900327       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5-1417k8s00000001-attachment/status 200 OK in 23 milliseconds
I0920 15:37:38.900739       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="285a29ff-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "latency"=23697264
I0920 15:37:38.900985       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="285a29ff-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5"
I0920 15:37:38.904442       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:38.909260       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:38.909508       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 15:37:38.913922       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022909462 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:38.914013       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:37:38.920530       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023509363 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:38.924546       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028166076 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:38.932617       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01854375 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:38.932832       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
... skipping 17 lines ...
I0920 15:37:39.012506       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e\" to node \"1417k8s00000001\"." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="2857fd7d-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e"
I0920 15:37:39.934923       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-y9dto3pe|1417k8s00000003" delay="1s"
I0920 15:37:39.935023       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:37:39.953984       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018914451 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:39.954016       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:37:39.954081       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 15:37:39.966514       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:39.966727       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="28590ce8-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=1121555133
I0920 15:37:39.966823       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="28590ce8-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=1121846233
I0920 15:37:39.966908       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="28590ce8-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=1141220885
I0920 15:37:39.966965       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.141386285 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" node="1417k8s00000003" result_code="succeeded"
I0920 15:37:39.966996       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0920 15:37:39.972317       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018146049 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:39.972397       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:37:39.972452       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-61797b4c-2183-455b-be8b-79b20c92eace:AttachDiskOptions{diskName: "pvc-61797b4c-2183-455b-be8b-79b20c92eace", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb:AttachDiskOptions{diskName: "pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb", lun: 1}]
I0920 15:37:39.972547       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-61797b4c-2183-455b-be8b-79b20c92eace:AttachDiskOptions{diskName: "pvc-61797b4c-2183-455b-be8b-79b20c92eace", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb:AttachDiskOptions{diskName: "pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb", lun: 1}])
I0920 15:37:39.973394       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb-1417k8s00000003-attachment/status 200 OK in 18 milliseconds
I0920 15:37:39.973986       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="28590ce8-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "latency"=19627553
I0920 15:37:39.980827       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-61797b4c-2183-455b-be8b-79b20c92eace-1417k8s00000003-attachment/status 200 OK in 26 milliseconds
I0920 15:37:39.981941       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:39.982000       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="2857f07b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=1131482859
I0920 15:37:39.982044       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="2857f07b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=1131735960
I0920 15:37:39.982083       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="2857f07b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=1163590546
I0920 15:37:39.982129       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.163736546 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-61797b4c-2183-455b-be8b-79b20c92eace" node="1417k8s00000003" result_code="succeeded"
I0920 15:37:39.982149       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 15:37:39.983041       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="2857f07b-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "latency"=28855978
... skipping 8 lines ...
I0920 15:37:40.031461       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017464248 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 15:37:40.031491       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 15:37:40.031508       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5:AttachDiskOptions{diskName: "pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e:AttachDiskOptions{diskName: "pvc-83cdeaff-ab63-4cc8-869b-94403969e88e", lun: 1}]
I0920 15:37:40.031563       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5:AttachDiskOptions{diskName: "pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e:AttachDiskOptions{diskName: "pvc-83cdeaff-ab63-4cc8-869b-94403969e88e", lun: 1}])
I0920 15:37:40.044512       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5-1417k8s00000001-attachment/status 200 OK in 30 milliseconds
I0920 15:37:40.044572       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e-1417k8s00000001-attachment/status 200 OK in 29 milliseconds
I0920 15:37:40.045844       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:40.045929       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="2857fd7d-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=1193458927
I0920 15:37:40.045979       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="2857fd7d-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=1193817128
I0920 15:37:40.046020       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="2857fd7d-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=1227189018
I0920 15:37:40.046065       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.227309018 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" node="1417k8s00000001" result_code="succeeded"
I0920 15:37:40.046084       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0920 15:37:40.046647       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:37:40.046934       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="285a29ff-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "latency"=32770788
I0920 15:37:40.047226       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="2857fd7d-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=32643388
I0920 15:37:40.047294       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="285a29ff-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "latency"=1197878638
I0920 15:37:40.047343       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="285a29ff-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "latency"=1198124340
I0920 15:37:40.047381       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="285a29ff-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" "latency"=1214436984
I0920 15:37:40.047419       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.214504784 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5" node="1417k8s00000001" result_code="succeeded"
... skipping 4 lines ...
I0920 15:37:40.059115       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name!=1417k8s00000001,disk.csi.azure.com/volume-name=pvc-83cdeaff-ab63-4cc8-869b-94403969e88e." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="291534ce-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e"
I0920 15:37:40.059223       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="291534ce-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=46600
I0920 15:37:40.059265       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="291534ce-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" "latency"=390901
I0920 15:37:40.059316       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.000480901 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e" node="1417k8s00000001" result_code="succeeded"
I0920 15:37:40.059335       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"1"}}
I0920 15:37:40.133769       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.161112635 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 15:37:40.133807       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-61797b4c-2183-455b-be8b-79b20c92eace:AttachDiskOptions{diskName: "pvc-61797b4c-2183-455b-be8b-79b20c92eace", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb:AttachDiskOptions{diskName: "pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 15:37:40.777217       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.745584216 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 15:37:40.777254       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5:AttachDiskOptions{diskName: "pvc-595ae4c8-f76d-40ee-888b-ac40c44f93a5", lun: 0} /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-83cdeaff-ab63-4cc8-869b-94403969e88e:AttachDiskOptions{diskName: "pvc-83cdeaff-ab63-4cc8-869b-94403969e88e", lun: 1}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 15:37:42.050538       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 25 milliseconds
I0920 15:37:42.060111       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 15:37:42.060389       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 15:37:44.067582       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 15:37:44.072417       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 15:37:44.072807       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 180 lines ...
I0920 15:38:50.157081       1 shared_state.go:588]  "msg"="Getting ranked list of nodes for creating AzVolumeAttachments" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.157287       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-61797b4c-2183-455b-be8b-79b20c92eace) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.157344       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-61797b4c-2183-455b-be8b-79b20c92eace)" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.157431       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/requested-role=Primary,disk.csi.azure.com/volume-name=pvc-61797b4c-2183-455b-be8b-79b20c92eace." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.157510       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.157576       1 shared_state.go:783]  "msg"="The list of zones to select nodes from is: westus2-1,westus2-2,westus2-3" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.157639       1 shared_state.go:795]  "msg"="failed to find zone annotations for primary node" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.157885       1 common.go:396]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0" "filter-plugin"="inter-pod affinity filter" "latency"=13000
I0920 15:38:50.157910       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod affinity filter): [1417k8s00000002 1417k8s00000000 k8s-master-14178977-vmss000000]" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.157954       1 common.go:513]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAntiAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0" "filter-plugin"="inter-pod anti-affinity filter" "latency"=8900
I0920 15:38:50.157973       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod anti-affinity filter): [1417k8s00000002 1417k8s00000000 k8s-master-14178977-vmss000000]" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.158002       1 common.go:584]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*podTolerationFilter).filter" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0" "filter-plugin"="pod toleration filter" "latency"=6600
I0920 15:38:50.158024       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (pod toleration filter): [1417k8s00000000 k8s-master-14178977-vmss000000 1417k8s00000002]" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
... skipping 65 lines ...
I0920 15:38:50.195893       1 shared_state.go:588]  "msg"="Getting ranked list of nodes for creating AzVolumeAttachments" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.196412       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-61797b4c-2183-455b-be8b-79b20c92eace) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.196738       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-61797b4c-2183-455b-be8b-79b20c92eace)" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.197655       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/requested-role=Primary,disk.csi.azure.com/volume-name=pvc-61797b4c-2183-455b-be8b-79b20c92eace." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.198524       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-e72272b0-bb5f-4ad8-a148-ed0c302bbadb) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.198874       1 shared_state.go:783]  "msg"="The list of zones to select nodes from is: westus2-1,westus2-2,westus2-3" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.200026       1 shared_state.go:795]  "msg"="failed to find zone annotations for primary node" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.200707       1 common.go:396]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0" "filter-plugin"="inter-pod affinity filter" "latency"=14300
I0920 15:38:50.200774       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod affinity filter): [1417k8s00000000 k8s-master-14178977-vmss000000 1417k8s00000002]" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.200851       1 common.go:513]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAntiAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0" "filter-plugin"="inter-pod anti-affinity filter" "latency"=10600
I0920 15:38:50.201182       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod anti-affinity filter): [1417k8s00000000 k8s-master-14178977-vmss000000 1417k8s00000002]" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
I0920 15:38:50.201285       1 common.go:584]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*podTolerationFilter).filter" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0" "filter-plugin"="pod toleration filter" "latency"=14100
I0920 15:38:50.201353       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (pod toleration filter): [1417k8s00000002 1417k8s00000000 k8s-master-14178977-vmss000000]" "disk.csi.azure.com/request-id"="52dcf1c6-38fa-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-61797b4c-2183-455b-be8b-79b20c92eace" "disk.csi/azure.com/pod-name"="azuredisk-3778/azuredisk-volume-tester-dm6wd-0"
... skipping 8617 lines ...
I0920 16:12:34.779212       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000002","volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"}
I0920 16:12:34.779441       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93-1417k8s00000002-attachment) detachment" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"
I0920 16:12:34.791896       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93-1417k8s00000002-attachment/status 200 OK in 12 milliseconds
I0920 16:12:34.799111       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=19555833
I0920 16:12:34.807921       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93-1417k8s00000002-attachment 200 OK in 8 milliseconds
I0920 16:12:34.808049       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93-1417k8s00000002-attachment)
I0920 16:12:34.811016       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:12:34.812066       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93) scheduled in 5m0s." "disk.csi.azure.com/request-id"="09a72be4-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"
I0920 16:12:34.821392       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93-1417k8s00000002-attachment/status 200 OK in 9 milliseconds
I0920 16:12:34.822066       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=9981717
I0920 16:12:34.822113       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"
I0920 16:12:34.822282       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93 from node 1417k8s00000002" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"
I0920 16:12:34.822303       1 azure_vmss_cache.go:332] Node 1417k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:12:34.824674       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:12:34.839937       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017592329 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:12:34.839965       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:12:34.839989       1 azure_vmss.go:204] Couldn't find VMSS VM with nodeName 1417k8s00000002, refreshing the cache(vmss: 1417k8s00, rg: kubetest-y9dto3pe)
I0920 16:12:34.937305       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.097255461 request="vmssvm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:12:35.939149       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-y9dto3pe|1417k8s00000002" delay="1s"
I0920 16:12:35.939214       1 azure_vmss_cache.go:332] Node 1417k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
... skipping 43 lines ...
I0920 16:12:56.214053       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93 from node 1417k8s00000002 successfully" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"
I0920 16:12:56.214254       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=21391881852
I0920 16:12:56.228464       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93-1417k8s00000002-attachment 200 OK in 13 milliseconds
I0920 16:12:56.229700       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=15380726
I0920 16:12:56.229757       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=21407558178
I0920 16:12:56.229799       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=21417748196
I0920 16:12:56.232528       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:12:56.232600       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=21424467406
I0920 16:12:56.232649       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="e79d85e6-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=21424615006
I0920 16:12:56.232740       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="09a231bc-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=21453322254
I0920 16:12:56.232787       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=21.453448855 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" node="1417k8s00000002" result_code="succeeded"
I0920 16:12:56.232805       1 utils.go:85] GRPC response: {}
I0920 16:12:56.475620       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
... skipping 11 lines ...
I0920 16:12:58.457395       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"}
I0920 16:12:58.457484       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93)
I0920 16:12:58.471830       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93/status 200 OK in 13 milliseconds
I0920 16:12:58.474018       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "disk.csi.azure.com/request-id"="17bf3214-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=16415028
I0920 16:12:58.474047       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93)
I0920 16:12:58.485196       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93 200 OK in 10 milliseconds
I0920 16:12:58.487369       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:12:58.498675       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0920 16:12:58.504348       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:12:58.505076       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93/status 200 OK in 18 milliseconds
I0920 16:12:58.505597       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "disk.csi.azure.com/request-id"="e5c900d0-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=19308932
I0920 16:12:58.505649       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "disk.csi.azure.com/request-id"="e5c900d0-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"
I0920 16:12:58.505724       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93)" "csi.storage.k8s.io/pv/name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "disk.csi.azure.com/request-id"="e5c900d0-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"
I0920 16:12:58.505754       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93)" "csi.storage.k8s.io/pv/name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "disk.csi.azure.com/request-id"="e5c900d0-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"
I0920 16:12:58.505805       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93." "csi.storage.k8s.io/pv/name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "disk.csi.azure.com/request-id"="e5c900d0-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93"
... skipping 8 lines ...
I0920 16:13:00.521733       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:13:00.537646       1 reflector.go:536] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Watch close - *v1beta2.AzVolume total 35 items received
I0920 16:13:00.540546       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=16528&timeoutSeconds=546&watch=true 200 OK in 2 milliseconds
I0920 16:13:01.905533       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:13:01.905533       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:13:01.915737       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:13:01.915860       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:13:02.447065       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:13:02.448154       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:13:02.448363       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:13:02.533314       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 10 milliseconds
I0920 16:13:02.539539       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:13:02.539972       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:13:03.733770       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.198926035 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:13:03.733803       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93
I0920 16:13:03.733861       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "disk.csi.azure.com/request-id"="e5c900d0-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=5227915483
I0920 16:13:03.755392       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:13:03.755465       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "disk.csi.azure.com/request-id"="17bf3214-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=5267829651
I0920 16:13:03.755507       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "disk.csi.azure.com/request-id"="17bf3214-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=5297930402
I0920 16:13:03.755520       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93) returned with <nil>
I0920 16:13:03.755571       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.298051202 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" result_code="succeeded"
I0920 16:13:03.755587       1 utils.go:85] GRPC response: {}
I0920 16:13:03.765474       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93 200 OK in 31 milliseconds
... skipping 16 lines ...
I0920 16:13:10.313736       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 16:13:10.313772       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c","parameters":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c","csi.storage.k8s.io/pvc/name":"pvc-cvlrf","csi.storage.k8s.io/pvc/namespace":"azuredisk-6003","maxshares":"2","skuname":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0920 16:13:10.314526       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:13:10.336573       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 21 milliseconds
I0920 16:13:10.337259       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:13:10.337441       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c)
I0920 16:13:10.343415       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:13:10.344889       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c/status 200 OK in 10 milliseconds
I0920 16:13:10.345621       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=11315621
I0920 16:13:10.345819       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:13:10.359405       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 16:13:10.359630       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.013407625 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 16:13:10.359668       1 cloudprovisioner.go:246] begin to create disk(pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c) account type(Premium_LRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(2)
I0920 16:13:10.385056       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.025315346 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:13:10.385099       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c StorageAccountType:Premium_LRS Size:10
I0920 16:13:10.528234       1 reflector.go:536] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Watch close - *v1.PersistentVolume total 32 items received
I0920 16:13:10.530655       1 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=16555&timeoutSeconds=578&watch=true 200 OK in 1 milliseconds
... skipping 5 lines ...
I0920 16:13:12.629381       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:13:12.629604       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:13:12.639782       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020327237 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:13:12.639814       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c StorageAccountType:Premium_LRS Size:10
I0920 16:13:12.639876       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c) account type(Premium_LRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c kubernetes.io-created-for-pvc-name:pvc-cvlrf kubernetes.io-created-for-pvc-namespace:azuredisk-6003]) successfully" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:13:12.639925       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=2293893569
I0920 16:13:12.653648       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:13:12.653736       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=2316033509
I0920 16:13:12.653816       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=2339500651
I0920 16:13:12.653899       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.339653652 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" result_code="succeeded"
I0920 16:13:12.653961       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c","csi.storage.k8s.io/pvc/name":"pvc-cvlrf","csi.storage.k8s.io/pvc/namespace":"azuredisk-6003","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"}}
I0920 16:13:12.656835       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c/status 200 OK in 16 milliseconds
I0920 16:13:12.657213       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=17232132
... skipping 2 lines ...
I0920 16:13:13.373001       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 16:13:13.373090       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000003","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c","csi.storage.k8s.io/pvc/name":"pvc-cvlrf","csi.storage.k8s.io/pvc/namespace":"azuredisk-6003","maxshares":"2","requestedsizegib":"10","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"}
I0920 16:13:13.373483       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000003,disk.csi.azure.com/volume-name!=pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:13:13.373830       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name!=1417k8s00000003,disk.csi.azure.com/volume-name=pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:13:13.379996       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 5 milliseconds
I0920 16:13:13.381655       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000003-attachment)
I0920 16:13:13.384070       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:13:13.392464       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:13:13.395785       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000003-attachment/status 200 OK in 13 milliseconds
I0920 16:13:13.397110       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=15089527
I0920 16:13:13.397163       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:13:13.432150       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.034791563 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:13:13.432212       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:13:13.453838       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.021547439 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 2 lines ...
I0920 16:13:13.454102       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c\" to node \"1417k8s00000003\"." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:13:14.454480       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-y9dto3pe|1417k8s00000003" delay="1s"
I0920 16:13:14.454574       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:13:14.470518       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015894329 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:13:14.470554       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:13:14.470609       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:13:14.480468       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:13:14.481344       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=1099591699
I0920 16:13:14.481409       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=1099802899
I0920 16:13:14.481457       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=1108108514
I0920 16:13:14.481512       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.108191514 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" node="1417k8s00000003" result_code="succeeded"
I0920 16:13:14.481531       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:13:14.487411       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000003-attachment/status 200 OK in 16 milliseconds
... skipping 3 lines ...
I0920 16:13:14.488112       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c:AttachDiskOptions{diskName: "pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c", lun: 0}]
I0920 16:13:14.488176       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c:AttachDiskOptions{diskName: "pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c", lun: 0}])
I0920 16:13:14.634095       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:13:14.639300       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:13:14.639483       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:13:14.659187       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.170966911 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 16:13:14.659233       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000003) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c:AttachDiskOptions{diskName: "pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 16:13:16.650377       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0920 16:13:16.688012       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 36 milliseconds
I0920 16:13:16.688684       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:13:18.693415       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:13:18.701509       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:13:18.701775       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 83 lines ...
I0920 16:13:51.773470       1 shared_state.go:330]  "msg"="Requested volumes for pod azuredisk-6003/azuredisk-volume-tester-vq9xl are now the following: Volumes: [pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c], Len: 1" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
I0920 16:13:51.773510       1 shared_state.go:588]  "msg"="Getting ranked list of nodes for creating AzVolumeAttachments" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
I0920 16:13:51.773898       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
I0920 16:13:51.774011       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c)" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
I0920 16:13:51.774223       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/requested-role=Primary,disk.csi.azure.com/volume-name=pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c." "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
I0920 16:13:51.774350       1 shared_state.go:783]  "msg"="The list of zones to select nodes from is: westus2-2" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
I0920 16:13:51.774461       1 shared_state.go:795]  "msg"="failed to find zone annotations for primary node" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
I0920 16:13:51.774820       1 common.go:396]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl" "filter-plugin"="inter-pod affinity filter" "latency"=10300
I0920 16:13:51.774923       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod affinity filter): [1417k8s00000003 1417k8s00000001]" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
I0920 16:13:51.775046       1 common.go:513]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAntiAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl" "filter-plugin"="inter-pod anti-affinity filter" "latency"=11400
I0920 16:13:51.775151       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod anti-affinity filter): [1417k8s00000003 1417k8s00000001]" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
I0920 16:13:51.775270       1 common.go:584]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*podTolerationFilter).filter" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl" "filter-plugin"="pod toleration filter" "latency"=9800
I0920 16:13:51.775308       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (pod toleration filter): [1417k8s00000003 1417k8s00000001]" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi/azure.com/pod-name"="azuredisk-6003/azuredisk-volume-tester-vq9xl"
... skipping 51 lines ...
I0920 16:13:52.935439       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000001-attachment/status 200 OK in 19 milliseconds
I0920 16:13:52.939023       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=22810595
I0920 16:13:52.943635       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 21 milliseconds
I0920 16:13:52.950281       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:13:52.950685       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:13:53.098954       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.167272299 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 16:13:53.098987       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c:AttachDiskOptions{diskName: "pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 16:13:54.960983       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 16:13:54.966021       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:13:54.966360       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:13:56.974095       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:13:56.980862       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:13:56.981191       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 63 lines ...
I0920 16:14:14.173075       1 round_trippers.go:553] PATCH https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000003-attachment/status 200 OK in 18 milliseconds
I0920 16:14:14.220826       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c/status 200 OK in 56 milliseconds
I0920 16:14:14.221690       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="44df0df5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=57822296
I0920 16:14:14.223239       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c)
I0920 16:14:14.273130       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000003-attachment 200 OK in 97 milliseconds
I0920 16:14:14.274071       1 shared_state.go:1028]  "msg"="Set deletion timestamp for AzVolumeAttachment (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000003-attachment)" "disk.csi.azure.com/request-id"="44dd4092-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:14.286969       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:14:14.288556       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c 200 OK in 64 milliseconds
I0920 16:14:14.328580       1 round_trippers.go:553] PATCH https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000001-attachment/status 200 OK in 53 milliseconds
I0920 16:14:14.340945       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000003-attachment/status 200 OK in 73 milliseconds
I0920 16:14:14.342188       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=74636752
I0920 16:14:14.342468       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:14.342853       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c from node 1417k8s00000003" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:14.343219       1 azure_vmss_cache.go:332] Node 1417k8s00000003 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:14:14.352624       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c/status 200 OK in 64 milliseconds
I0920 16:14:14.355717       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:14:14.356372       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=68337731
I0920 16:14:14.356634       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:14.356930       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c)" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:14.357285       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c)" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:14.357559       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:14.358788       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015107351 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 10 lines ...
I0920 16:14:14.393549       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c from node 1417k8s00000001" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:14.393915       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:14:14.397494       1 round_trippers.go:553] PATCH https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000001-attachment/status 200 OK in 12 milliseconds
I0920 16:14:14.399031       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=42093642
I0920 16:14:14.399212       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000003-attachment)
I0920 16:14:14.399378       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000001-attachment)
I0920 16:14:14.399779       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:14:14.440475       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.046332857 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:14:14.440508       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:14:14.447160       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.087705397 request="vmssvm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:14:15.124021       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 16:14:15.129668       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:14:15.130286       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 31 lines ...
I0920 16:14:27.202624       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:14:27.214133       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 10 milliseconds
I0920 16:14:27.214581       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:14:27.761834       1 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0920 16:14:27.761860       1 utils.go:79] GRPC request: {}
I0920 16:14:27.762114       1 utils.go:85] GRPC response: {"ready":{"value":true}}
E0920 16:14:29.163507       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="44df0df5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=14851981926
E0920 16:14:29.163861       1 crdprovisioner.go:368]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="44df0df5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=14999915126
I0920 16:14:29.163957       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c) returned with context deadline exceeded
I0920 16:14:29.164145       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=15.000609728 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" result_code="failed"
E0920 16:14:29.164229       1 utils.go:83] GRPC error: context deadline exceeded
I0920 16:14:29.217507       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0920 16:14:29.224589       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:14:29.225167       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:14:30.163951       1 utils.go:78] GRPC call: /csi.v1.Controller/DeleteVolume
I0920 16:14:30.163988       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"}
I0920 16:14:30.164381       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c)
... skipping 4 lines ...
I0920 16:14:30.751164       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c:pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c]) returned with <nil>
I0920 16:14:30.751248       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 1417k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c:pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c]
I0920 16:14:30.751367       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c) succeeded
I0920 16:14:30.751420       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c from node 1417k8s00000001 successfully" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:30.751513       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=16358080283
I0920 16:14:30.764473       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000001-attachment 200 OK in 12 milliseconds
I0920 16:14:30.767605       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:14:30.767915       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=16368266513
I0920 16:14:30.768576       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=17017852
I0920 16:14:30.768693       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=16375262637
I0920 16:14:30.768894       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="378656b7-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=16396682909
I0920 16:14:30.818692       1 replica.go:170]  "msg"="Checking for replicas to be created after garbage collection." "disk.csi.azure.com/request-id"="9bfbf80c-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-9a137c97-2fc3-4370-acde-4f0a62a1f541"
I0920 16:14:30.819057       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="9bfbf80c-38fe-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "disk.csi.azure.com/volume-name"="pvc-9a137c97-2fc3-4370-acde-4f0a62a1f541" "latency"=230600
... skipping 3 lines ...
I0920 16:14:31.315537       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=15.851102118 request="vmssvm_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="detach_disk" error_code=""
I0920 16:14:31.315571       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000003) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c:pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c]) returned with <nil>
I0920 16:14:31.315644       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 1417k8s00000003: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c:pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c]
I0920 16:14:31.315795       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c) succeeded
I0920 16:14:31.315916       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c from node 1417k8s00000003 successfully" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c"
I0920 16:14:31.316193       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=16973332509
I0920 16:14:31.327691       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:14:31.328100       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=16928422454
I0920 16:14:31.329512       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c-1417k8s00000003-attachment 200 OK in 12 milliseconds
I0920 16:14:31.331305       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=15038947
I0920 16:14:31.331376       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=16988576856
I0920 16:14:31.331423       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000003" "disk.csi.azure.com/request-id"="20a3295e-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=17063896810
I0920 16:14:31.359791       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.031307297 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:14:31.908645       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:14:31.909140       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:14:31.918453       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:14:31.918749       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:14:32.447905       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:14:32.448990       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:14:32.449357       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:14:33.249861       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0920 16:14:33.256334       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:14:33.256700       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:14:35.265593       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 16:14:35.273664       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:14:35.274307       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:14:36.522216       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.162347541 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:14:36.522248       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c
I0920 16:14:36.522502       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=5193866939
I0920 16:14:36.536736       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c 200 OK in 13 milliseconds
I0920 16:14:36.538083       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:14:36.538562       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="4e689b72-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=6373013803
I0920 16:14:36.538650       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="4e689b72-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=6373866006
I0920 16:14:36.538719       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c) returned with <nil>
I0920 16:14:36.538849       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=6.374359907 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" result_code="succeeded"
I0920 16:14:36.538872       1 utils.go:85] GRPC response: {}
I0920 16:14:36.539703       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "disk.csi.azure.com/request-id"="1ed063e5-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-199b9786-e2f0-424e-bf28-b75f8c66ae8c" "latency"=17136453
... skipping 15 lines ...
I0920 16:14:41.308612       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:14:41.315333       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:14:41.316195       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:14:41.319173       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 22 milliseconds
I0920 16:14:41.323220       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:14:41.323256       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c)
I0920 16:14:41.328897       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:14:41.334958       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c/status 200 OK in 10 milliseconds
I0920 16:14:41.336251       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=11799540
I0920 16:14:41.336485       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:14:41.337742       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:14:41.360921       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 16:14:41.361167       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.024236983 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 16:14:41.361205       1 cloudprovisioner.go:246] begin to create disk(pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0920 16:14:41.390267       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.0290074 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:14:41.390317       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c StorageAccountType:StandardSSD_LRS Size:10
I0920 16:14:43.320851       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:14:43.327302       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
... skipping 4 lines ...
I0920 16:14:43.726470       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c kubernetes.io-created-for-pvc-name:pvc1-azuredisk-volume-tester-vdd6n-0 kubernetes.io-created-for-pvc-namespace:azuredisk-8423]) successfully" "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:14:43.726630       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=2389857107
I0920 16:14:43.746172       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c/status 200 OK in 18 milliseconds
I0920 16:14:43.746804       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=19735368
I0920 16:14:43.746862       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=2410172976
I0920 16:14:43.746902       1 workflow.go:155]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=2422494118
I0920 16:14:43.747585       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:14:43.747650       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=2424176525
I0920 16:14:43.747693       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=2451100917
I0920 16:14:43.747738       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.451225517 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" result_code="succeeded"
I0920 16:14:43.747757       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c","csi.storage.k8s.io/pvc/name":"pvc1-azuredisk-volume-tester-vdd6n-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8423","requestedsizegib":"10","skuname":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"}}
I0920 16:14:44.336739       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 16:14:44.336776       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c","csi.storage.k8s.io/pvc/name":"pvc1-azuredisk-volume-tester-vdd6n-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-8423","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"}
I0920 16:14:44.337450       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000001,disk.csi.azure.com/volume-name!=pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:14:44.351620       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 13 milliseconds
I0920 16:14:44.353878       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c-1417k8s00000001-attachment)
I0920 16:14:44.362529       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c-1417k8s00000001-attachment/status 200 OK in 7 milliseconds
I0920 16:14:44.363989       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:14:44.364439       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=9848934
I0920 16:14:44.364479       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:14:44.387928       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023332281 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:14:44.387994       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:14:44.406561       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.018532264 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:14:44.406589       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
... skipping 8 lines ...
I0920 16:14:45.489553       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:14:45.521700       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.031673989 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:14:45.521735       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:14:45.522045       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:14:45.532578       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c-1417k8s00000001-attachment/status 200 OK in 10 milliseconds
I0920 16:14:45.533951       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=11823133
I0920 16:14:45.534663       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:14:45.534776       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=1180778731
I0920 16:14:45.535012       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=1181160631
I0920 16:14:45.535061       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=1197827390
I0920 16:14:45.535241       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.19804649 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" node="1417k8s00000001" result_code="succeeded"
I0920 16:14:45.535267       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:14:45.540033       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01767695 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 5 lines ...
I0920 16:14:45.544548       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000001,disk.csi.azure.com/volume-name!=pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="5793588c-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:14:45.544796       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="5793588c-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=48300
I0920 16:14:45.544998       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="5793588c-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=467101
I0920 16:14:45.545096       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.000792403 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" node="1417k8s00000001" result_code="succeeded"
I0920 16:14:45.545115       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:14:45.719241       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.178223602 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 16:14:45.719279       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c:AttachDiskOptions{diskName: "pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 16:14:47.339528       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:14:47.344431       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:14:47.345038       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:14:49.351615       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:14:49.355417       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:14:49.356240       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 250 lines ...
I0920 16:16:39.267050       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c-1417k8s00000001-attachment) detachment" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:16:39.283105       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c-1417k8s00000001-attachment/status 200 OK in 15 milliseconds
I0920 16:16:39.283639       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=16517325
I0920 16:16:39.299269       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c-1417k8s00000001-attachment 200 OK in 15 milliseconds
I0920 16:16:39.301091       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c) scheduled in 5m0s." "disk.csi.azure.com/request-id"="9b613c81-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:16:39.301857       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c-1417k8s00000001-attachment)
I0920 16:16:39.311413       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:16:39.313268       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c-1417k8s00000001-attachment/status 200 OK in 11 milliseconds
I0920 16:16:39.313837       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=12520919
I0920 16:16:39.313881       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:16:39.314000       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c from node 1417k8s00000001" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:16:39.314022       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:16:39.348659       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.034597352 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 35 lines ...
I0920 16:16:55.688163       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c:pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c]) returned with <nil>
I0920 16:16:55.688203       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 1417k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c:pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c]
I0920 16:16:55.688259       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c) succeeded
I0920 16:16:55.688369       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c from node 1417k8s00000001 successfully" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:16:55.688419       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=16374460359
I0920 16:16:55.703371       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c-1417k8s00000001-attachment 200 OK in 14 milliseconds
I0920 16:16:55.705895       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:16:55.706245       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=17743078
I0920 16:16:55.706295       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=16392353938
I0920 16:16:55.706334       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=16405055558
I0920 16:16:55.706390       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=16404448457
I0920 16:16:55.706429       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="56db263f-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=16404589357
I0920 16:16:55.706462       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="9b5c0ae4-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=16439449109
... skipping 17 lines ...
I0920 16:17:01.147240       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"}
I0920 16:17:01.147325       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c)
I0920 16:17:01.164725       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c/status 200 OK in 17 milliseconds
I0920 16:17:01.165528       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="a866ba55-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=18074542
I0920 16:17:01.165759       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c)
I0920 16:17:01.176622       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c 200 OK in 10 milliseconds
I0920 16:17:01.178675       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:17:01.190120       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c/status 200 OK in 9 milliseconds
I0920 16:17:01.193412       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=13275215
I0920 16:17:01.193800       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:17:01.193863       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:17:01.194048       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c)" "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:17:01.194317       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c)" "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:17:01.194505       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c"
I0920 16:17:01.194671       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=593101
I0920 16:17:01.217047       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.022289304 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:17:01.913195       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:17:01.914273       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:17:01.923464       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:17:01.923564       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:17:02.325920       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:17:02.334466       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 16:17:02.334796       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:17:02.450789       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:17:02.451994       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:17:02.452090       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 4 lines ...
I0920 16:17:06.351339       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:17:06.352092       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:17:06.367518       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.15042264 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:17:06.367547       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c
I0920 16:17:06.367604       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=5172847044
I0920 16:17:06.393226       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c 200 OK in 25 milliseconds
I0920 16:17:06.394851       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:17:06.396549       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="a866ba55-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=5215844063
I0920 16:17:06.396610       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="a866ba55-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=5249181442
I0920 16:17:06.396625       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c) returned with <nil>
I0920 16:17:06.396699       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.249313542 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" result_code="succeeded"
I0920 16:17:06.396720       1 utils.go:85] GRPC response: {}
I0920 16:17:06.397248       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "disk.csi.azure.com/request-id"="550b2f3a-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-9e455b10-187b-4f8b-a609-a8ab3461ce9c" "latency"=29393905
... skipping 32 lines ...
I0920 16:17:22.623920       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:17:22.623952       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225)
I0920 16:17:22.642669       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225/status 200 OK in 19 milliseconds
I0920 16:17:22.643117       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=20857041
I0920 16:17:22.643156       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:17:22.643297       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-2) is reset as empty since disk(pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:17:22.645222       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:17:22.660891       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 16:17:22.661054       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017592935 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 16:17:22.661093       1 cloudprovisioner.go:246] begin to create disk(pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone() maxShares(2)
I0920 16:17:22.677818       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016620132 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:17:22.677911       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225 StorageAccountType:StandardSSD_ZRS Size:10
I0920 16:17:24.497699       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 16:17:24.505426       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:17:24.506080       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:17:24.900128       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.222155924 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:17:24.935781       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.03559297 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:17:24.935902       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225 StorageAccountType:StandardSSD_ZRS Size:10
I0920 16:17:24.936256       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225 kubernetes.io-created-for-pvc-name:pvc-7fwlv kubernetes.io-created-for-pvc-namespace:azuredisk-9561]) successfully" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:17:24.936451       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=2293089563
I0920 16:17:24.945852       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225/status 200 OK in 8 milliseconds
I0920 16:17:24.949207       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:17:24.949654       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=2325546426
I0920 16:17:24.950387       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=2363356999
I0920 16:17:24.950460       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.363433299 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" result_code="succeeded"
I0920 16:17:24.950489       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225","csi.storage.k8s.io/pvc/name":"pvc-7fwlv","csi.storage.k8s.io/pvc/namespace":"azuredisk-9561","maxshares":"2","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"}}
I0920 16:17:24.950350       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=13828927
I0920 16:17:24.950839       1 azvolume.go:166]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=2307598591
... skipping 13 lines ...
I0920 16:17:27.030638       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name!=1417k8s00000000,disk.csi.azure.com/volume-name=pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:17:27.050096       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 18 milliseconds
I0920 16:17:27.050491       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225-1417k8s00000000-attachment)
I0920 16:17:27.064342       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225-1417k8s00000000-attachment/status 200 OK in 13 milliseconds
I0920 16:17:27.064813       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=14162969
I0920 16:17:27.064914       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:17:27.069139       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:17:27.085705       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020617201 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:17:27.085789       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:17:27.125069       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.039234891 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:17:27.125186       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:17:27.125370       1 cloudprovisioner.go:398]  "msg"="GetDiskLun returned: -1. Initiating attaching volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225\" to node \"1417k8s00000000\"." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:17:27.125493       1 cloudprovisioner.go:412]  "msg"="Trying to attach volume \"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225\" to node \"1417k8s00000000\"." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
... skipping 2 lines ...
I0920 16:17:27.761358       1 utils.go:85] GRPC response: {"ready":{"value":true}}
I0920 16:17:28.125870       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="attach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-y9dto3pe|1417k8s00000000" delay="1s"
I0920 16:17:28.125971       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:17:28.142385       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01634868 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:17:28.142415       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:17:28.142461       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:17:28.154650       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:17:28.154771       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=1104133593
I0920 16:17:28.154835       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=1104367494
I0920 16:17:28.154881       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=1125601897
I0920 16:17:28.154929       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.125727699 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" node="1417k8s00000000" result_code="succeeded"
I0920 16:17:28.154953       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:17:28.161251       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225-1417k8s00000000-attachment/status 200 OK in 18 milliseconds
I0920 16:17:28.161552       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=18898392
I0920 16:17:28.165092       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.02258941 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:17:28.165134       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:17:28.165149       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000000: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225:AttachDiskOptions{diskName: "pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225", lun: 0}]
I0920 16:17:28.165207       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225:AttachDiskOptions{diskName: "pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225", lun: 0}])
I0920 16:17:28.346800       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.181523487 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 16:17:28.346839       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000000) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225:AttachDiskOptions{diskName: "pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 16:17:28.538201       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:17:28.554922       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 14 milliseconds
I0920 16:17:28.555713       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:17:30.563079       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:17:30.568733       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:17:30.569151       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 109 lines ...
I0920 16:18:16.853730       1 shared_state.go:330]  "msg"="Requested volumes for pod azuredisk-9561/azuredisk-volume-tester-cjqfk are now the following: Volumes: [pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225], Len: 1" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
I0920 16:18:16.853762       1 shared_state.go:588]  "msg"="Getting ranked list of nodes for creating AzVolumeAttachments" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
I0920 16:18:16.853956       1 shared_state.go:734]  "msg"="Number of requested replicas for Azvolume (pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225) is: 1. Max replica count is: 1." "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
I0920 16:18:16.854024       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225)" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
I0920 16:18:16.854201       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/requested-role=Primary,disk.csi.azure.com/volume-name=pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225." "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
I0920 16:18:16.854265       1 shared_state.go:783]  "msg"="The list of zones to select nodes from is: westus2-1,westus2-2,westus2-3" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
I0920 16:18:16.854325       1 shared_state.go:795]  "msg"="failed to find zone annotations for primary node" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
I0920 16:18:16.854558       1 common.go:396]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk" "filter-plugin"="inter-pod affinity filter" "latency"=14400
I0920 16:18:16.854601       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod affinity filter): [1417k8s00000000 k8s-master-14178977-vmss000000 1417k8s00000002]" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
I0920 16:18:16.854816       1 common.go:513]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*interPodAntiAffinityFilter).filter" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk" "filter-plugin"="inter-pod anti-affinity filter" "latency"=93300
I0920 16:18:16.854865       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (inter-pod anti-affinity filter): [1417k8s00000000 k8s-master-14178977-vmss000000 1417k8s00000002]" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
I0920 16:18:16.854918       1 common.go:584]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*podTolerationFilter).filter" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk" "filter-plugin"="pod toleration filter" "latency"=10800
I0920 16:18:16.854951       1 shared_state.go:652]  "msg"="Filtered node list from filter plugin (pod toleration filter): [1417k8s00000000 k8s-master-14178977-vmss000000 1417k8s00000002]" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi/azure.com/pod-name"="azuredisk-9561/azuredisk-volume-tester-cjqfk"
... skipping 78 lines ...
I0920 16:18:18.064365       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=12245823
I0920 16:18:18.069217       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017343232 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:18:18.069246       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:18:18.069303       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225:AttachDiskOptions{diskName: "pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225", lun: 0}]
I0920 16:18:18.069421       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225:AttachDiskOptions{diskName: "pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225", lun: 0}])
I0920 16:18:18.245324       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.175832424 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 16:18:18.245360       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225:AttachDiskOptions{diskName: "pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 16:18:18.952896       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:18:18.968102       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:18:18.968909       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:18:19.876711       1 reflector.go:536] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 35 items received
I0920 16:18:19.879293       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=17769&timeout=6m49s&timeoutSeconds=409&watch=true 200 OK in 2 milliseconds
I0920 16:18:20.989434       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 19 milliseconds
... skipping 85 lines ...
I0920 16:18:38.863442       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"}
I0920 16:18:38.863763       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225)
I0920 16:18:38.880998       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225/status 200 OK in 16 milliseconds
I0920 16:18:38.883796       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="e2a519fc-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=19727847
I0920 16:18:38.883827       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225)
I0920 16:18:38.892230       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225 200 OK in 8 milliseconds
I0920 16:18:38.893710       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:18:38.920194       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225/status 200 OK in 23 milliseconds
I0920 16:18:38.925412       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:18:38.926002       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=29889271
I0920 16:18:38.926037       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:18:38.926185       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225)" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:18:38.926271       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225)" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:18:38.926352       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:18:38.939731       1 round_trippers.go:553] PATCH https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225-1417k8s00000000-attachment/status 200 OK in 12 milliseconds
... skipping 5 lines ...
I0920 16:18:38.964379       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225 from node 1417k8s00000000" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:18:38.964401       1 azure_vmss_cache.go:332] Node 1417k8s00000000 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:18:38.972046       1 round_trippers.go:553] PATCH https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225-1417k8s00000001-attachment/status 200 OK in 8 milliseconds
I0920 16:18:38.972995       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=46807011
I0920 16:18:38.973020       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225-1417k8s00000000-attachment)
I0920 16:18:38.973067       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225-1417k8s00000001-attachment)
I0920 16:18:38.975532       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:18:39.000073       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.035627585 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:18:39.000107       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:18:39.142535       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 16 milliseconds
I0920 16:18:39.150459       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:18:39.153253       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:18:40.000680       1 batch.go:224] "cloud-provider-azure: Delayed processing of batch due to start delay" type="batch" operation="detach_disk" key="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e|kubetest-y9dto3pe|1417k8s00000000" delay="1s"
... skipping 21 lines ...
I0920 16:18:51.230926       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:18:51.239772       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:18:51.240356       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:18:53.250458       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0920 16:18:53.258084       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:18:53.258556       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
E0920 16:18:53.864122       1 conditionwaiter.go:50]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="e2a519fc-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=14967080517
E0920 16:18:53.864967       1 crdprovisioner.go:368]  "msg"="Workflow completed with an error." "error"="rpc error: code = Internal desc = [context deadline exceeded]" "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="e2a519fc-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=15000909997
I0920 16:18:53.865339       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225) returned with context deadline exceeded
I0920 16:18:53.865465       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=15.001615999 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" result_code="failed"
E0920 16:18:53.865546       1 utils.go:83] GRPC error: context deadline exceeded
I0920 16:18:55.264984       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:18:55.269820       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:18:55.271552       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:18:55.874766       1 reflector.go:536] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolumeAttachment total 51 items received
I0920 16:18:55.881430       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=18099&timeout=5m4s&timeoutSeconds=304&watch=true 200 OK in 6 milliseconds
I0920 16:18:57.276943       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 8 lines ...
I0920 16:18:59.814673       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=25.263544192 request="vmssvm_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="detach_disk" error_code=""
I0920 16:18:59.814713       1 azure_controller_vmss.go:210] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - detach disk(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225:pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225]) returned with <nil>
I0920 16:18:59.814753       1 azure_controller_common.go:417] azuredisk - successfully detached disks from node 1417k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225:pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225]
I0920 16:18:59.814802       1 azure_controller_common.go:378] azureDisk - detach disk(pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225, /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225) succeeded
I0920 16:18:59.814853       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225 from node 1417k8s00000001 successfully" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:18:59.814900       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=26404748349
I0920 16:18:59.827733       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:18:59.827903       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=20854691106
I0920 16:18:59.828693       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225-1417k8s00000001-attachment 200 OK in 12 milliseconds
I0920 16:18:59.829831       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=14862827
I0920 16:18:59.829879       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=26419756776
I0920 16:18:59.829916       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="d5867160-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=26434332902
I0920 16:19:00.329568       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=20.313176555 request="vmssvm_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="detach_disk" error_code=""
... skipping 3 lines ...
I0920 16:19:00.329759       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225 from node 1417k8s00000000 successfully" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225"
I0920 16:19:00.329818       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=21365536350
I0920 16:19:00.342638       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225-1417k8s00000000-attachment 200 OK in 12 milliseconds
I0920 16:19:00.343014       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=13128924
I0920 16:19:00.343055       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=21378804474
I0920 16:19:00.343076       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b7d3fc35-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=21391414004
I0920 16:19:00.345059       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:19:00.345148       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Replica" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=21371945457
I0920 16:19:00.366174       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020910339 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:19:01.302191       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:19:01.307071       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:19:01.307229       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:19:01.917072       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 16 lines ...
I0920 16:19:05.331885       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:19:05.338269       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:19:05.338812       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:19:05.504854       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.138607084 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:19:05.504891       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225
I0920 16:19:05.505058       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="b52e271b-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=5159692124
I0920 16:19:05.521950       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:19:05.523714       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="f119f5ff-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=2404964526
I0920 16:19:05.523919       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225 200 OK in 17 milliseconds
I0920 16:19:05.524205       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "disk.csi.azure.com/request-id"="f119f5ff-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" "latency"=2405956428
I0920 16:19:05.524424       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225) returned with <nil>
I0920 16:19:05.524692       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.406914231 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-021f235b-25d0-4b94-b31f-7e3cd0c8c225" result_code="succeeded"
I0920 16:19:05.524870       1 utils.go:85] GRPC response: {}
... skipping 12 lines ...
I0920 16:19:12.348433       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 16:19:12.348681       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7","parameters":{"csi.storage.k8s.io/pv/name":"pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7","csi.storage.k8s.io/pvc/name":"pvc-78c5d","csi.storage.k8s.io/pvc/namespace":"azuredisk-6039","skuname":"StandardSSD_ZRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}}]}
I0920 16:19:12.349069       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:19:12.370266       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 21 milliseconds
I0920 16:19:12.372729       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:19:12.372814       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7)
I0920 16:19:12.379995       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:19:12.384606       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:19:12.386734       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7/status 200 OK in 12 milliseconds
I0920 16:19:12.387108       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=12802240
I0920 16:19:12.387166       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:19:12.387342       1 cloudprovisioner.go:207]  "msg"="diskZone(westus2-1) is reset as empty since disk(pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7) is ZRS(StandardSSD_ZRS)" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:19:12.400908       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 16:19:12.401013       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.013287641 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 16:19:12.401044       1 cloudprovisioner.go:246] begin to create disk(pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone() maxShares(0)
I0920 16:19:12.427976       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.026886384 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:19:12.428022       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7 StorageAccountType:StandardSSD_ZRS Size:10
I0920 16:19:13.385075       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 12 milliseconds
I0920 16:19:13.392710       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:19:13.392979       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:19:14.616589       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.188499358 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:19:14.644985       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028328989 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:19:14.645032       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7 StorageAccountType:StandardSSD_ZRS Size:10
I0920 16:19:14.645109       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7) account type(StandardSSD_ZRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7 kubernetes.io-created-for-pvc-name:pvc-78c5d kubernetes.io-created-for-pvc-namespace:azuredisk-6039]) successfully" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:19:14.645158       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=2257878676
I0920 16:19:14.656691       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:19:14.656774       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=2283829557
I0920 16:19:14.656824       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=2307796432
I0920 16:19:14.656872       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.307860732 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" result_code="succeeded"
I0920 16:19:14.656892       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}},{"segments":{"topology.disk.csi.azure.com/zone":"westus2-3"}},{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7","csi.storage.k8s.io/pvc/name":"pvc-78c5d","csi.storage.k8s.io/pvc/namespace":"azuredisk-6039","requestedsizegib":"10","skuname":"StandardSSD_ZRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"}}
I0920 16:19:14.667785       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7/status 200 OK in 22 milliseconds
I0920 16:19:14.677027       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=31644099
... skipping 10 lines ...
I0920 16:19:16.822421       1 shared_state.go:483]  "msg"="Storing pod azuredisk-volume-tester-dzdz6-9b89f548f-sjp9l and claim [] to podToClaimsMap map." 
I0920 16:19:16.960820       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 16:19:16.960857       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000001","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4","mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7","csi.storage.k8s.io/pvc/name":"pvc-78c5d","csi.storage.k8s.io/pvc/namespace":"azuredisk-6039","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"}
I0920 16:19:16.961451       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000001,disk.csi.azure.com/volume-name!=pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:19:16.974100       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 12 milliseconds
I0920 16:19:16.974551       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7-1417k8s00000001-attachment)
I0920 16:19:16.975055       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:19:16.978405       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:19:16.981379       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7-1417k8s00000001-attachment/status 200 OK in 11 milliseconds
I0920 16:19:16.981769       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=13321741
I0920 16:19:16.981877       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:19:17.002841       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020697065 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:19:17.002910       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:19:17.018470       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015481948 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 9 lines ...
I0920 16:19:18.129956       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:19:18.150020       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019848645 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:19:18.150054       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:19:18.150147       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:19:18.158119       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7-1417k8s00000001-attachment/status 200 OK in 7 milliseconds
I0920 16:19:18.158776       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=8306118
I0920 16:19:18.161642       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:19:18.161964       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=1187250855
I0920 16:19:18.162195       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=1187638956
I0920 16:19:18.162267       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=1200945198
I0920 16:19:18.162360       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.201082198 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" node="1417k8s00000001" result_code="succeeded"
I0920 16:19:18.162429       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:19:18.169741       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 5 lines ...
I0920 16:19:18.170921       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:19:18.212711       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.062459241 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:19:18.212747       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:19:18.212765       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7:AttachDiskOptions{diskName: "pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7", lun: 0}]
I0920 16:19:18.212934       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7:AttachDiskOptions{diskName: "pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7", lun: 0}])
I0920 16:19:18.382304       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.169248281 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 16:19:18.382342       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7:AttachDiskOptions{diskName: "pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 16:19:19.429665       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:19:19.435266       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:19:19.436006       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:19:21.439713       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:19:21.444788       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:19:21.445406       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 132 lines ...
I0920 16:20:10.357373       1 pod.go:99]  "msg"="Pod azuredisk-6039/azuredisk-volume-tester-dzdz6-9b89f548f-gnhvf has 1 volumes. Volumes: [pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7]" "disk.csi.azure.com/request-id"="192dd577-3900-11ed-bff3-000d3af50766" "disk.csi/azure.com/pod-name"="azuredisk-6039/azuredisk-volume-tester-dzdz6-9b89f548f-gnhvf"
I0920 16:20:10.357709       1 pod.go:125]  "msg"="No need to create replica attachment for volume (pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7). Replica controller is responsible for it" "disk.csi.azure.com/request-id"="192dd577-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi/azure.com/pod-name"="azuredisk-6039/azuredisk-volume-tester-dzdz6-9b89f548f-gnhvf"
I0920 16:20:10.447115       1 pod.go:65] AzDiskControllerManager "msg"="No need to reconcile. Pod was deleted." "controller"="pod" "disk.csi/azure.com/pod-name"="azuredisk-volume-tester-dzdz6-9b89f548f-sjp9l" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0920 16:20:11.810988       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:20:11.817755       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:20:11.819843       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:20:13.528360       1 node_availability.go:59] AzDiskControllerManager "msg"="Node is now available. Will requeue failed replica creation requests." "controller"="nodeavailability" "disk.csi.azure.com/node-name"="1417k8s00000000" "namespace"="azure-disk-csi" "partition"="csi-azuredisk-controller"
I0920 16:20:13.528735       1 shared_state.go:1064]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="1b11c4cf-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).tryCreateFailedReplicas" "latency"=14500
I0920 16:20:13.830546       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 9 milliseconds
I0920 16:20:13.836557       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:20:13.836903       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:20:15.844806       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 7 milliseconds
I0920 16:20:15.849950       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
... skipping 17 lines ...
I0920 16:20:23.779880       1 crdprovisioner.go:926]  "msg"="Requesting AzVolumeAttachment (pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7-1417k8s00000001-attachment) detachment" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:20:23.786537       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7-1417k8s00000001-attachment/status 200 OK in 5 milliseconds
I0920 16:20:23.789381       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).detachVolume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=9415820
I0920 16:20:23.798980       1 replica.go:151]  "msg"="Garbage collection of AzVolumeAttachments for AzVolume (pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7) scheduled in 5m0s." "disk.csi.azure.com/request-id"="2130e59d-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:20:23.803145       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7-1417k8s00000001-attachment 200 OK in 13 milliseconds
I0920 16:20:23.803455       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7-1417k8s00000001-attachment)
I0920 16:20:23.810081       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:20:23.811733       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7-1417k8s00000001-attachment/status 200 OK in 12 milliseconds
I0920 16:20:23.812251       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=13256928
I0920 16:20:23.812342       1 attach_detach.go:332]  "msg"="Detaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:20:23.812468       1 cloudprovisioner.go:470]  "msg"="Trying to detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7 from node 1417k8s00000001" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:20:23.812491       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:20:23.829081       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016546434 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 20 lines ...
I0920 16:20:27.950380       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 16:20:27.951215       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:20:29.955176       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0920 16:20:29.960207       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:20:29.960505       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:20:31.920080       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:20:31.920180       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:20:31.920200       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:20:31.926480       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:20:31.964253       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:20:31.971746       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:20:31.972124       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:20:32.453151       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 18 lines ...
I0920 16:20:40.137616       1 cloudprovisioner.go:480]  "msg"="detach volume /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7 from node 1417k8s00000001 successfully" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:20:40.137780       1 cloudprovisioner.go:460]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=16325351101
I0920 16:20:40.161578       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7-1417k8s00000001-attachment 200 OK in 23 milliseconds
I0920 16:20:40.165008       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=27035133
I0920 16:20:40.165064       1 attach_detach.go:338]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach.func3" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=16352653334
I0920 16:20:40.165104       1 workflow.go:155]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerDetach" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=16366132562
I0920 16:20:40.165107       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:20:40.165182       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=16361647852
I0920 16:20:40.165221       1 crdprovisioner.go:950]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="f95a47c1-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForDetach" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=16361809653
I0920 16:20:40.165247       1 crdprovisioner.go:837]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="212dfb48-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).UnpublishVolume" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=16385482502
I0920 16:20:40.165278       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=16.385600202 request="azuredisk_csi_driver_controller_unpublish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" node="1417k8s00000001" result_code="succeeded"
I0920 16:20:40.165289       1 utils.go:85] GRPC response: {}
I0920 16:20:40.193313       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
... skipping 18 lines ...
I0920 16:20:48.915832       1 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"}
I0920 16:20:48.916032       1 controllerserver_v2.go:200] deleting disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7)
I0920 16:20:48.930468       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7/status 200 OK in 13 milliseconds
I0920 16:20:48.933955       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="30297da9-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=17683431
I0920 16:20:48.933985       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7)
I0920 16:20:48.943149       1 round_trippers.go:553] DELETE https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7 200 OK in 8 milliseconds
I0920 16:20:48.945688       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:20:48.957641       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7/status 200 OK in 15 milliseconds
I0920 16:20:48.959763       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:20:48.966366       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=24078843
I0920 16:20:48.966420       1 azvolume.go:251]  "msg"="Deleting Volume..." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:20:48.966724       1 shared_state.go:931]  "msg"="AzVolumeAttachment clean up requested by azvolume-controller for AzVolume (pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7)" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:20:48.966953       1 azure_disk_utils.go:1014]  "msg"="Getting AzVolumeAttachment list for volume (pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7)" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:20:48.967162       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/volume-name=pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7"
I0920 16:20:48.967231       1 shared_state.go:929]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*SharedState).cleanUpAzVolumeAttachmentByVolume" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=508300
... skipping 6 lines ...
I0920 16:20:52.130522       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:20:54.125506       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.137901377 request="disks_delete" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:20:54.125539       1 azure_managedDiskController.go:303] azureDisk - deleted a managed disk: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7
I0920 16:20:54.125853       1 cloudprovisioner.go:328]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=5158174712
I0920 16:20:54.143937       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 12 milliseconds
I0920 16:20:54.150478       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7 200 OK in 24 milliseconds
I0920 16:20:54.152925       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:20:54.153213       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="30297da9-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=5206967021
I0920 16:20:54.153479       1 crdprovisioner.go:368]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="30297da9-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).DeleteVolume" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=5237188775
I0920 16:20:54.153671       1 controllerserver_v2.go:202] delete disk(/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7) returned with <nil>
I0920 16:20:54.153909       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=5.237845276 request="azuredisk_csi_driver_controller_delete_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" result_code="succeeded"
I0920 16:20:54.154090       1 utils.go:85] GRPC response: {}
I0920 16:20:54.156345       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerDelete.func4" "csi.storage.k8s.io/pv/name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "disk.csi.azure.com/request-id"="f69a80eb-38ff-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-4a6fdb74-3851-4f77-aa56-8a3c64529ed7" "latency"=30324178
... skipping 20 lines ...
I0920 16:21:01.859380       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 16:21:01.859413       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2","topology.kubernetes.io/zone":"westus2-2"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66","parameters":{"csi.storage.k8s.io/pv/name":"pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0920 16:21:01.859716       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "disk.csi.azure.com/request-id"="37e0807c-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66"
I0920 16:21:01.898798       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 38 milliseconds
I0920 16:21:01.899540       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "disk.csi.azure.com/request-id"="37e0807c-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66"
I0920 16:21:01.899559       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66)
I0920 16:21:01.905293       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:21:01.908593       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66/status 200 OK in 11 milliseconds
I0920 16:21:01.908851       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "disk.csi.azure.com/request-id"="37e0807c-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=12278621
I0920 16:21:01.908892       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "disk.csi.azure.com/request-id"="37e0807c-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66"
I0920 16:21:01.920418       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:21:01.920506       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:21:01.922257       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 16:21:01.922418       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.013321323 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 16:21:01.922508       1 cloudprovisioner.go:246] begin to create disk(pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) selectedAvailabilityZone(westus2-2) maxShares(0)
I0920 16:21:01.927383       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:21:01.927470       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:21:01.946567       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.023970441 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:01.946718       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66 StorageAccountType:StandardSSD_LRS Size:10
I0920 16:21:02.210090       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 2 milliseconds
I0920 16:21:02.221394       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 16:21:02.221777       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:21:02.453918       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:21:02.454988       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:21:02.457350       1 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:21:04.177253       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.230439693 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:04.195053       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.01772903 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:04.195099       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66 StorageAccountType:StandardSSD_LRS Size:10
I0920 16:21:04.195388       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66 kubernetes.io-created-for-pvc-name:pvc-azuredisk kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "disk.csi.azure.com/request-id"="37e0807c-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66"
I0920 16:21:04.195523       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "disk.csi.azure.com/request-id"="37e0807c-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=2286470488
I0920 16:21:04.209010       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:21:04.209052       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66/status 200 OK in 13 milliseconds
I0920 16:21:04.209545       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "disk.csi.azure.com/request-id"="37e0807c-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=2309839727
I0920 16:21:04.209843       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "disk.csi.azure.com/request-id"="37e0807c-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=2349925196
I0920 16:21:04.210036       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.350246896 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" result_code="succeeded"
I0920 16:21:04.210066       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-2"}}],"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66","csi.storage.k8s.io/pvc/name":"pvc-azuredisk","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66"}}
I0920 16:21:04.212123       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "disk.csi.azure.com/request-id"="37e0807c-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=16539328
... skipping 7 lines ...
I0920 16:21:04.883206       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000001,disk.csi.azure.com/volume-name!=pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="39add3c4-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66"
I0920 16:21:04.894927       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 10 milliseconds
I0920 16:21:04.895525       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66-1417k8s00000001-attachment)
I0920 16:21:04.901040       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66-1417k8s00000001-attachment/status 200 OK in 10 milliseconds
I0920 16:21:04.902720       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="39add3c4-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=12678921
I0920 16:21:04.902772       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="39add3c4-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66"
I0920 16:21:04.904955       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:21:04.923096       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019541933 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:04.923193       1 azure_vmss_cache.go:332] Node 1417k8s00000001 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:21:04.939500       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.016238028 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:04.939531       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:21:04.939558       1 azure_vmss.go:204] Couldn't find VMSS VM with nodeName 1417k8s00000001, refreshing the cache(vmss: 1417k8s00, rg: kubetest-y9dto3pe)
I0920 16:21:05.082930       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.143335344 request="vmssvm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
... skipping 7 lines ...
I0920 16:21:06.109228       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66-1417k8s00000001-attachment/status 200 OK in 10 milliseconds
I0920 16:21:06.109858       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="39add3c4-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=10891019
I0920 16:21:06.114791       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015939727 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:06.114814       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:21:06.114828       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000001: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66:AttachDiskOptions{diskName: "pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66", lun: 0}]
I0920 16:21:06.114873       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66:AttachDiskOptions{diskName: "pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66", lun: 0}])
I0920 16:21:06.115798       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:21:06.116096       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="39add3c4-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=1220241275
I0920 16:21:06.116183       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="39add3c4-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=1220676575
I0920 16:21:06.116265       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000001" "disk.csi.azure.com/request-id"="39add3c4-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" "latency"=1233246597
I0920 16:21:06.116367       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.233398997 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" node="1417k8s00000001" result_code="succeeded"
I0920 16:21:06.116414       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:21:06.123531       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
... skipping 4 lines ...
I0920 16:21:06.124520       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.000653801 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66" node="1417k8s00000001" result_code="succeeded"
I0920 16:21:06.124537       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:21:06.259385       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:21:06.263847       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:21:06.264705       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:21:06.303297       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.188375321 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 16:21:06.303560       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000001) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66:AttachDiskOptions{diskName: "pvc-60e7ae48-82f5-4949-b7e1-4ff775950d66", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 16:21:08.273655       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 8 milliseconds
I0920 16:21:08.280664       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:21:08.281443       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:21:10.287398       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:21:10.294340       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 6 milliseconds
I0920 16:21:10.294626       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 92 lines ...
I0920 16:21:43.977107       1 utils.go:78] GRPC call: /csi.v1.Controller/CreateVolume
I0920 16:21:43.977169       1 utils.go:79] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1","topology.kubernetes.io/zone":"westus2-1"}}]},"capacity_range":{"required_bytes":107374182400},"name":"pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af","parameters":{"csi.storage.k8s.io/pv/name":"pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0920 16:21:43.978122       1 crdprovisioner.go:297]  "msg"="Creating AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "disk.csi.azure.com/request-id"="50fb4157-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"
I0920 16:21:44.005371       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes 201 Created in 26 milliseconds
I0920 16:21:44.006136       1 crdprovisioner.go:305]  "msg"="Successfully created AzVolume CRI" "csi.storage.k8s.io/pv/name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "disk.csi.azure.com/request-id"="50fb4157-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"
I0920 16:21:44.006348       1 conditionwatcher.go:113] Adding a condition function for azvolume (pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af)
I0920 16:21:44.010026       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:21:44.011195       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af/status 200 OK in 10 milliseconds
I0920 16:21:44.012141       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate" "csi.storage.k8s.io/pv/name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "disk.csi.azure.com/request-id"="50fb4157-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=11628645
I0920 16:21:44.012438       1 azvolume.go:159]  "msg"="Creating Volume..." "csi.storage.k8s.io/pv/name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "disk.csi.azure.com/request-id"="50fb4157-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"
I0920 16:21:44.030193       1 azure_diskclient.go:139] Received error in disk.get.request: resourceID: /subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af' under resource group 'kubetest-y9dto3pe' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
I0920 16:21:44.030497       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.017730222 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code="ResourceNotFound"
I0920 16:21:44.030669       1 cloudprovisioner.go:246] begin to create disk(pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(100) selectedAvailabilityZone(westus2-1) maxShares(0)
I0920 16:21:44.058754       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.028014051 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:44.058814       1 azure_managedDiskController.go:92] azureDisk - creating new managed Name:pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af StorageAccountType:StandardSSD_LRS Size:100
I0920 16:21:44.556258       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 4 milliseconds
I0920 16:21:44.561092       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 3 milliseconds
I0920 16:21:44.561400       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:21:46.283284       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.224413929 request="disks_create_or_update" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:46.303943       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.020589535 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:46.303981       1 azure_managedDiskController.go:266] azureDisk - created new MD Name:pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af StorageAccountType:StandardSSD_LRS Size:100
I0920 16:21:46.304048       1 cloudprovisioner.go:311]  "msg"="create disk(pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af) account type(StandardSSD_LRS) rg(kubetest-y9dto3pe) location() size(100) tags(map[kubernetes.io-created-for-pv-name:pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-win-0 kubernetes.io-created-for-pvc-namespace:default]) successfully" "csi.storage.k8s.io/pv/name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "disk.csi.azure.com/request-id"="50fb4157-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"
I0920 16:21:46.304108       1 cloudprovisioner.go:142]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "disk.csi.azure.com/request-id"="50fb4157-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CloudProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=2291424144
I0920 16:21:46.321599       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:21:46.321982       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "disk.csi.azure.com/request-id"="50fb4157-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=2315220550
I0920 16:21:46.322173       1 crdprovisioner.go:219]  "msg"="Workflow completed with success." "csi.storage.k8s.io/pv/name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "disk.csi.azure.com/request-id"="50fb4157-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).CreateVolume" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=2344123210
I0920 16:21:46.322358       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=2.344516612 request="azuredisk_csi_driver_controller_create_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" result_code="succeeded"
I0920 16:21:46.322388       1 utils.go:85] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"westus2-1"}}],"capacity_bytes":107374182400,"content_source":{"Type":{"Volume":{}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"}}
I0920 16:21:46.323723       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumes/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af/status 200 OK in 19 milliseconds
I0920 16:21:46.324236       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAzVolume).triggerCreate.func3" "csi.storage.k8s.io/pv/name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "disk.csi.azure.com/request-id"="50fb4157-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=20025534
... skipping 5 lines ...
I0920 16:21:47.096888       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 16:21:47.096927       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"}
I0920 16:21:47.097910       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000002,disk.csi.azure.com/volume-name!=pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="52d7415d-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"
I0920 16:21:47.110377       1 round_trippers.go:553] POST https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments 201 Created in 12 milliseconds
I0920 16:21:47.113781       1 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af-1417k8s00000002-attachment)
I0920 16:21:47.125528       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af-1417k8s00000002-attachment/status 200 OK in 12 milliseconds
I0920 16:21:47.127632       1 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:21:47.129479       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="52d7415d-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=16431429
I0920 16:21:47.129738       1 attach_detach.go:182]  "msg"="Attaching volume" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="52d7415d-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"
I0920 16:21:47.149701       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.019579734 request="disks_get" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:47.149796       1 azure_vmss_cache.go:332] Node 1417k8s00000002 has joined the cluster since the last VM cache refresh, refreshing the cache
I0920 16:21:47.165726       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015872527 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:47.165783       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
... skipping 7 lines ...
I0920 16:21:48.198162       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.015115226 request="vm_list" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="" error_code=""
I0920 16:21:48.198196       1 azure_backoff.go:101] VirtualMachinesClient.List(kubetest-y9dto3pe) success
I0920 16:21:48.198212       1 azure_controller_common.go:306] azuredisk - trying to attach disks to node 1417k8s00000002: map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af:AttachDiskOptions{diskName: "pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af", lun: 0}]
I0920 16:21:48.198305       1 azure_controller_vmss.go:109] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af:AttachDiskOptions{diskName: "pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af", lun: 0}])
I0920 16:21:48.199760       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azvolumeattachments/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af-1417k8s00000002-attachment/status 200 OK in 16 milliseconds
I0920 16:21:48.202238       1 azure_disk_utils.go:1072]  "msg"="Workflow completed with success." "caller"="sigs.k8s.io/azuredisk-csi-driver/pkg/controller.(*ReconcileAttachDetach).triggerAttach.func3.4" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="52d7415d-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azureutils.UpdateCRIWithRetry" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=19280633
I0920 16:21:48.203931       1 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:21:48.204002       1 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="52d7415d-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=1089416086
I0920 16:21:48.204048       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="52d7415d-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=1090295588
I0920 16:21:48.204090       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="52d7415d-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=1106532716
I0920 16:21:48.204243       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=1.106935717 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" node="1417k8s00000002" result_code="succeeded"
I0920 16:21:48.204365       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:21:48.210828       1 utils.go:78] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0920 16:21:48.210859       1 utils.go:79] GRPC request: {"node_id":"1417k8s00000002","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"}
I0920 16:21:48.211368       1 azure_disk_utils.go:1043]  "msg"="Label selector is: disk.csi.azure.com/node-name=1417k8s00000002,disk.csi.azure.com/volume-name!=pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="53812988-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"
I0920 16:21:48.211521       1 crdprovisioner.go:724]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="53812988-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).waitForLun" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=47300
I0920 16:21:48.211564       1 crdprovisioner.go:484]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="53812988-3900-11ed-bff3-000d3af50766" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).PublishVolume" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=505401
I0920 16:21:48.211614       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.000587301 request="azuredisk_csi_driver_controller_publish_volume" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="disk.csi.azure.com" volumeid="/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" node="1417k8s00000002" result_code="succeeded"
I0920 16:21:48.211632       1 utils.go:85] GRPC response: {"publish_context":{"LUN":"0"}}
I0920 16:21:48.362660       1 azure_metrics.go:114] "Observed Request Latency" latency_seconds=0.164271485 request="vmssvm_updateasync" resource_group="kubetest-y9dto3pe" subscription_id="0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e" source="attach_disk" error_code=""
I0920 16:21:48.362708       1 azure_controller_vmss.go:121] azureDisk - update(kubetest-y9dto3pe): vm(1417k8s00000002) - attach disk list(map[/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af:AttachDiskOptions{diskName: "pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af", lun: 0}], %!s(*retry.Error=<nil>)) returned with %!v(MISSING)
I0920 16:21:48.579623       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:21:48.586326       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 5 milliseconds
I0920 16:21:48.586819       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
I0920 16:21:50.614618       1 round_trippers.go:553] GET https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 26 milliseconds
I0920 16:21:50.645587       1 round_trippers.go:553] PUT https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/csi-azuredisk-controller 200 OK in 30 milliseconds
I0920 16:21:50.645827       1 leaderelection.go:278] successfully renewed lease kube-system/csi-azuredisk-controller
... skipping 157 lines ...
I0920 15:19:47.405956    6344 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=1620&timeout=8m53s&timeoutSeconds=533&watch=true 200 OK in 2 milliseconds
I0920 15:19:47.406050    6344 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=1621&timeout=7m59s&timeoutSeconds=479&watch=true 200 OK in 2 milliseconds
I0920 15:19:47.501349    6344 shared_informer.go:285] caches populated
I0920 15:19:47.501349    6344 azuredisk_v2.go:191] driver userAgent: disk.csi.azure.com/latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 e2e-test
I0920 15:19:47.502450    6344 azure_disk_utils.go:543] reading cloud config from secret kube-system/azure-cloud-provider
I0920 15:19:47.509152    6344 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 3 milliseconds
I0920 15:19:47.510826    6344 azure_disk_utils.go:550] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0920 15:19:47.510826    6344 azure_disk_utils.go:555] could not read cloud config from secret kube-system/azure-cloud-provider
I0920 15:19:47.510826    6344 azure_disk_utils.go:565] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0920 15:19:47.522848    6344 azure_disk_utils.go:573] read cloud config from file: C:\k\azure.json successfully
I0920 15:19:47.530237    6344 azure_auth.go:245] Using AzurePublicCloud environment
I0920 15:19:47.531142    6344 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I0920 15:19:47.531176    6344 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
I0920 15:19:47.531224    6344 azure_auth.go:113] azure: User Assigned MSI ID is client ID. Resource ID parsing error: %+vparsing failed for 0f481d95-6b3d-4086-b24e-9836129cae7a. Invalid resource Id format
I0920 15:19:47.535069    6344 azure_diskclient.go:68] Azure DisksClient using API version: 2021-04-01
I0920 15:19:47.543830    6344 azure.go:1011] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333333, Bucket: 80
I0920 15:19:47.547921    6344 safe_mounter_windows.go:406] using CSIProxyMounterV1, API Versions filesystem: v1, disk: v1, volume: v1
I0920 15:19:47.549915    6344 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0920 15:19:47.549915    6344 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0920 15:19:47.549915    6344 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
... skipping 307 lines ...
I0920 15:31:22.756091    6344 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-d94df931-324b-4db3-ac29-91a858763a58-1417k8s00000002-attachment)
I0920 15:31:22.756091    6344 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64-1417k8s00000002-attachment)
I0920 15:31:22.756091    6344 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3-1417k8s00000002-attachment)
I0920 15:31:27.162614    6344 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0920 15:31:27.162614    6344 utils.go:79] GRPC request: {}
I0920 15:31:27.162614    6344 utils.go:85] GRPC response: {"ready":{"value":true}}
I0920 15:31:35.727733    6344 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:35.727733    6344 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="483154f2-38f9-11ed-b98f-00155d727351" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-d94df931-324b-4db3-ac29-91a858763a58" "latency"=12971663000
I0920 15:31:35.727733    6344 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="483154f2-38f9-11ed-b98f-00155d727351" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12971941600
I0920 15:31:35.744183    6344 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:35.744183    6344 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="483154f2-38f9-11ed-b991-00155d727351" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64" "latency"=12988113100
I0920 15:31:35.744183    6344 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="483154f2-38f9-11ed-b991-00155d727351" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12988391700
I0920 15:31:35.745261    6344 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:35.745261    6344 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="483154f2-38f9-11ed-b990-00155d727351" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3" "latency"=12988480500
I0920 15:31:35.745261    6344 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="483154f2-38f9-11ed-b990-00155d727351" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=12989469800
I0920 15:31:42.464533    6344 nodeserver_v2.go:211] NodeStageVolume: formatting 8 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3\globalmount with mount options([])
I0920 15:31:42.464533    6344 nodeserver_v2.go:211] NodeStageVolume: formatting 7 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64\globalmount with mount options([])
I0920 15:31:42.467442    6344 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-d94df931-324b-4db3-ac29-91a858763a58\globalmount with mount options([])
I0920 15:31:47.531075    6344 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 258 lines ...
I0920 15:33:45.534608    6344 utils.go:85] GRPC response: {"usage":[{"available":10677252096,"total":10719588352,"unit":1,"used":42336256}]}
I0920 15:33:45.537276    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0920 15:33:45.537276    6344 utils.go:79] GRPC request: {}
I0920 15:33:45.537276    6344 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0920 15:33:45.540096    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0920 15:33:45.540211    6344 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-d94df931-324b-4db3-ac29-91a858763a58","volume_path":"c:\\var\\lib\\kubelet\\pods\\b449437b-0a13-480a-b0aa-ceaee52b2715\\volumes\\kubernetes.io~csi\\pvc-d94df931-324b-4db3-ac29-91a858763a58\\mount"}
E0920 15:33:46.664778    6344 utils.go:83] GRPC error: GetVolumeIDFromMount(c:\var\lib\kubelet\pods\b449437b-0a13-480a-b0aa-ceaee52b2715\volumes\kubernetes.io~csi\pvc-d94df931-324b-4db3-ac29-91a858763a58\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\b449437b-0a13-480a-b0aa-ceaee52b2715\volumes\kubernetes.io~csi\pvc-d94df931-324b-4db3-ac29-91a858763a58\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\b449437b-0a13-480a-b0aa-ceaee52b2715\volumes\kubernetes.io~csi\pvc

-d94df931-324b-4db3-ac29-91a858763a58\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\b449437b-0a13-480a-b0aa-ceaee ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...858763a58\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0920 15:33:46.988625    6344 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-94f4a732-b60b-4b20-a776-0ce59c1a35d3\globalmount successfully
I0920 15:33:46.988625    6344 utils.go:85] GRPC response: {}
I0920 15:33:47.013141    6344 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-d94df931-324b-4db3-ac29-91a858763a58\globalmount successfully
I0920 15:33:47.013141    6344 utils.go:85] GRPC response: {}
I0920 15:33:47.061945    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0920 15:33:47.061945    6344 utils.go:79] GRPC request: {}
... skipping 154 lines ...
I0920 15:34:45.281058    6344 utils.go:85] GRPC response: {"usage":[{"available":10677248000,"total":10719588352,"unit":1,"used":42340352}]}
I0920 15:34:45.288079    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0920 15:34:45.288167    6344 utils.go:79] GRPC request: {}
I0920 15:34:45.288167    6344 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0920 15:34:45.290524    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeGetVolumeStats
I0920 15:34:45.290594    6344 utils.go:79] GRPC request: {"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64","volume_path":"c:\\var\\lib\\kubelet\\pods\\36eade71-c8c1-4a37-a74c-f132e51b1e9d\\volumes\\kubernetes.io~csi\\pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64\\mount"}
E0920 15:34:46.742583    6344 utils.go:83] GRPC error: GetVolumeIDFromMount(c:\var\lib\kubelet\pods\36eade71-c8c1-4a37-a74c-f132e51b1e9d\volumes\kubernetes.io~csi\pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64\mount) failed with error: rpc error: code = Unknown desc = error getting the volume for the mount c:\var\lib\kubelet\pods\36eade71-c8c1-4a37-a74c-f132e51b1e9d\volumes\kubernetes.io~csi\pvc-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64\mount, internal error error getting volume from mount. cmd: (Get-Item -Path Get-Item : Cannot find path 'C:\var\lib\kubelet\pods\36eade71-c8c1-4a37-a74c-f132e51b1e9d\volumes\kubernetes.io~csi\pvc

-df1f88b4-8bb4-4c64-acf6-de6d31e6cb64\mount' because it does not exist.
At line:1 char:2
+ (Get-Item -Path c:\var\lib\kubelet\pods\36eade71-c8c1-4a37-a74c-f132e ...
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\var\lib\kube...d31e6cb64\mount:String) [Get-Item], ItemNotFoundExcep 
   tion
... skipping 37 lines ...
+ ... d : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand).Target
+                                                                  ~
Unexpected token ')' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression
 
, error: exit status 1, response: <nil>
I0920 15:34:47.214346    6344 nodeserver_v2.go:262] NodeUnstageVolume: unmount \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-d94df931-324b-4db3-ac29-91a858763a58\globalmount successfully
I0920 15:34:47.214346    6344 utils.go:85] GRPC response: {}
I0920 15:34:47.269256    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0920 15:34:47.269256    6344 utils.go:79] GRPC request: {}
I0920 15:34:47.269256    6344 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0920 15:34:47.278694    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
... skipping 521 lines ...
I0920 15:47:23.054247    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0920 15:47:23.054247    6344 utils.go:79] GRPC request: {}
I0920 15:47:23.054247    6344 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0920 15:47:23.056376    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0920 15:47:23.056376    6344 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-053a4853-62eb-4086-a83e-dcdcf4beb846\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-053a4853-62eb-4086-a83e-dcdcf4beb846","csi.storage.k8s.io/pvc/name":"pvc-dqz24","csi.storage.k8s.io/pvc/namespace":"azuredisk-4070","enableAsyncAttach":"false","enableBursting":"true","enableasyncattach":"false","perfProfile":"Basic","requestedsizegib":"1024","skuname":"Premium_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com","userAgent":"azuredisk-e2e-test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-053a4853-62eb-4086-a83e-dcdcf4beb846"}
I0920 15:47:23.056376    6344 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-053a4853-62eb-4086-a83e-dcdcf4beb846-1417k8s00000002-attachment)
I0920 15:47:26.562863    6344 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:47:26.562863    6344 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="849391b0-38fb-11ed-b991-00155d727351" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-053a4853-62eb-4086-a83e-dcdcf4beb846" "latency"=3505847500
I0920 15:47:26.562863    6344 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="849391b0-38fb-11ed-b991-00155d727351" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=3506511300
I0920 15:47:27.159407    6344 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0920 15:47:27.159407    6344 utils.go:79] GRPC request: {}
I0920 15:47:27.159939    6344 utils.go:85] GRPC response: {"ready":{"value":true}}
I0920 15:47:30.988359    6344 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-053a4853-62eb-4086-a83e-dcdcf4beb846\globalmount with mount options([])
... skipping 479 lines ...
I0920 16:08:39.801443    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0920 16:08:39.801443    6344 utils.go:79] GRPC request: {}
I0920 16:08:39.801443    6344 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0920 16:08:39.803343    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0920 16:08:39.803343    6344 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-9a137c97-2fc3-4370-acde-4f0a62a1f541\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["barrier=1","acl"]}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-9a137c97-2fc3-4370-acde-4f0a62a1f541","csi.storage.k8s.io/pvc/name":"pvc-x8ssn","csi.storage.k8s.io/pvc/namespace":"azuredisk-1431","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com","tags":"disk=test"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-9a137c97-2fc3-4370-acde-4f0a62a1f541"}
I0920 16:08:39.803343    6344 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-9a137c97-2fc3-4370-acde-4f0a62a1f541-1417k8s00000002-attachment)
I0920 16:08:42.306271    6344 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:08:42.306387    6344 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="7d93b21c-38fe-11ed-b991-00155d727351" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-9a137c97-2fc3-4370-acde-4f0a62a1f541" "latency"=2502929600
I0920 16:08:42.306465    6344 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="7d93b21c-38fe-11ed-b991-00155d727351" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=2503123600
I0920 16:08:42.506072    6344 reflector.go:536] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: Watch close - *v1beta2.AzVolume total 69 items received
I0920 16:08:42.507421    6344 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumes?allowWatchBookmarks=true&resourceVersion=15350&timeout=7m0s&timeoutSeconds=420&watch=true 200 OK in 0 milliseconds
I0920 16:08:46.931449    6344 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-9a137c97-2fc3-4370-acde-4f0a62a1f541\globalmount with mount options([barrier=1 acl])
I0920 16:08:47.994263    6344 azuredisk_v2.go:549]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-20T16:08:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="827587e9-38fe-11ed-b991-00155d727351"
... skipping 102 lines ...
I0920 16:11:47.996837    6344 azuredisk_v2.go:549]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-20T16:11:47Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="edbfbe75-38fe-11ed-b991-00155d727351"
I0920 16:11:48.007233    6344 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/1417k8s00000002/status 200 OK in 6 milliseconds
I0920 16:11:48.009753    6344 azuredisk_v2.go:536]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="edbfbe75-38fe-11ed-b991-00155d727351" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12372600
I0920 16:11:48.115417    6344 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:11:48.115417    6344 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:11:48.164596    6344 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:11:48.164935    6344 conditionwatcher.go:187] condition result: succeeded: false, error: <nil>
I0920 16:11:48.355222    6344 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 16:11:49.038390    6344 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:11:49.038488    6344 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="eb3e5ed8-38fe-11ed-b991-00155d727351" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93" "latency"=5244934500
I0920 16:11:49.038488    6344 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="eb3e5ed8-38fe-11ed-b991-00155d727351" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=5244995900
I0920 16:11:53.283552    6344 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-8d7cd405-f37e-4e8d-9683-3ef37dacbe93\globalmount with mount options([])
I0920 16:11:57.152763    6344 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0920 16:11:57.152763    6344 utils.go:79] GRPC request: {}
I0920 16:11:57.152763    6344 utils.go:85] GRPC response: {"ready":{"value":true}}
... skipping 249 lines ...
I0920 16:21:57.067724    6344 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0920 16:21:57.067724    6344 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-win-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"100","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af"}
I0920 16:21:57.068324    6344 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af-1417k8s00000002-attachment)
I0920 16:21:57.147335    6344 utils.go:78] GRPC call: /csi.v1.Identity/Probe
I0920 16:21:57.147335    6344 utils.go:79] GRPC request: {}
I0920 16:21:57.147459    6344 utils.go:85] GRPC response: {"ready":{"value":true}}
I0920 16:21:58.458704    6344 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 16:21:58.459265    6344 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="58c8aaf7-3900-11ed-b991-00155d727351" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af" "latency"=1390382400
I0920 16:21:58.459354    6344 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="58c8aaf7-3900-11ed-b991-00155d727351" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=1391096000
I0920 16:22:02.667575    6344 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-5dcac915-c8a9-49f9-a835-bad4d5d737af\globalmount with mount options([])
I0920 16:22:17.985496    6344 azuredisk_v2.go:549]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-20T16:22:17Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="65406274-3900-11ed-b991-00155d727351"
I0920 16:22:17.992693    6344 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/1417k8s00000002/status 200 OK in 6 milliseconds
I0920 16:22:17.997181    6344 azuredisk_v2.go:536]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="1417k8s00000002" "disk.csi.azure.com/request-id"="65406274-3900-11ed-b991-00155d727351" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=11685300
... skipping 64 lines ...
I0920 15:19:54.316932    6388 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azvolumeattachments?allowWatchBookmarks=true&resourceVersion=1621&timeout=5m45s&timeoutSeconds=345&watch=true 200 OK in 1 milliseconds
I0920 15:19:54.317394    6388 round_trippers.go:553] GET https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/azdrivernodes?allowWatchBookmarks=true&resourceVersion=1950&timeout=5m59s&timeoutSeconds=359&watch=true 200 OK in 2 milliseconds
I0920 15:19:54.418376    6388 shared_informer.go:285] caches populated
I0920 15:19:54.418376    6388 azuredisk_v2.go:191] driver userAgent: disk.csi.azure.com/latest-v2-626365bd77ad6040ec2f2e9da1a786ebdb1eaa03 e2e-test
I0920 15:19:54.418376    6388 azure_disk_utils.go:543] reading cloud config from secret kube-system/azure-cloud-provider
I0920 15:19:54.430385    6388 round_trippers.go:553] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/secrets/azure-cloud-provider 404 Not Found in 10 milliseconds
I0920 15:19:54.430890    6388 azure_disk_utils.go:550] InitializeCloudFromSecret: failed to get cloud config from secret kube-system/azure-cloud-provider: failed to get secret kube-system/azure-cloud-provider: secrets "azure-cloud-provider" not found
I0920 15:19:54.430890    6388 azure_disk_utils.go:555] could not read cloud config from secret kube-system/azure-cloud-provider
I0920 15:19:54.430890    6388 azure_disk_utils.go:565] use default AZURE_CREDENTIAL_FILE env var: C:\k\azure.json
I0920 15:19:54.441384    6388 azure_disk_utils.go:573] read cloud config from file: C:\k\azure.json successfully
I0920 15:19:54.442386    6388 azure_auth.go:245] Using AzurePublicCloud environment
I0920 15:19:54.442386    6388 azure_auth.go:96] azure: using managed identity extension to retrieve access token
I0920 15:19:54.442386    6388 azure_auth.go:102] azure: using User Assigned MSI ID to retrieve access token
I0920 15:19:54.442877    6388 azure_auth.go:113] azure: User Assigned MSI ID is client ID. Resource ID parsing error: %+vparsing failed for 0f481d95-6b3d-4086-b24e-9836129cae7a. Invalid resource Id format
I0920 15:19:54.443381    6388 azure_diskclient.go:68] Azure DisksClient using API version: 2021-04-01
I0920 15:19:54.443893    6388 azure.go:1011] attach/detach disk operation rate limiter configuration - Enabled: true QPS: 1.333333, Bucket: 80
I0920 15:19:54.445891    6388 safe_mounter_windows.go:406] using CSIProxyMounterV1, API Versions filesystem: v1, disk: v1, volume: v1
I0920 15:19:54.445891    6388 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0920 15:19:54.445891    6388 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0920 15:19:54.445891    6388 driver.go:81] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
... skipping 53 lines ...
I0920 15:20:34.655364    6388 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0920 15:20:34.655364    6388 utils.go:79] GRPC request: {}
I0920 15:20:34.655364    6388 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0920 15:20:34.658369    6388 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0920 15:20:34.658369    6388 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-ec43aca1-3abb-4b3c-a901-38a027d61779\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-ec43aca1-3abb-4b3c-a901-38a027d61779","csi.storage.k8s.io/pvc/name":"pvc-hq7cf","csi.storage.k8s.io/pvc/namespace":"azuredisk-1884","requestedsizegib":"10","skuname":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-ec43aca1-3abb-4b3c-a901-38a027d61779"}
I0920 15:20:34.659371    6388 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-ec43aca1-3abb-4b3c-a901-38a027d61779-1417k8s00000000-attachment)
I0920 15:20:42.036147    6388 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:20:42.036147    6388 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="c5e5a9b1-38f7-11ed-8eaf-00155d1e80c6" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-ec43aca1-3abb-4b3c-a901-38a027d61779" "latency"=7376762000
I0920 15:20:42.036147    6388 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="c5e5a9b1-38f7-11ed-8eaf-00155d1e80c6" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7376762000
I0920 15:20:47.839776    6388 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-ec43aca1-3abb-4b3c-a901-38a027d61779\globalmount with mount options([])
I0920 15:20:54.319684    6388 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:20:54.319784    6388 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
I0920 15:20:54.319784    6388 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 286 lines ...
I0920 15:31:28.141982    6388 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-2b876417-4866-4173-a2b6-9a5f2335716c-1417k8s00000000-attachment)
I0920 15:31:28.141982    6388 utils.go:79] GRPC request: {"publish_context":{"LUN":"2"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2\\globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2","csi.storage.k8s.io/pvc/name":"test-volume-3-azuredisk-volume-tester-wftzv-0","csi.storage.k8s.io/pvc/namespace":"azuredisk-6803","requestedsizegib":"10","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2"}
I0920 15:31:28.142484    6388 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2-1417k8s00000000-attachment)
I0920 15:31:30.054162    6388 azuredisk_v2.go:549]  "msg"="Updating heartbeat" "LastHeartbeatTime"="2022-09-20T15:31:30Z" "ReadyForVolumeAllocation"=true "StatusMessage"="Driver node healthy." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="4c8af8ba-38f9-11ed-8eb0-00155d1e80c6"
I0920 15:31:30.066622    6388 round_trippers.go:553] PUT https://10.0.0.1:443/apis/disk.csi.azure.com/v1beta2/namespaces/azure-disk-csi/azdrivernodes/1417k8s00000000/status 200 OK in 12 milliseconds
I0920 15:31:30.067157    6388 azuredisk_v2.go:536]  "msg"="Workflow completed with success." "disk.csi.azure.com/namespace"="azure-disk-csi" "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="4c8af8ba-38f9-11ed-8eb0-00155d1e80c6" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/azuredisk.(*DriverV2).updateAzDriverNodeHeartbeat" "latency"=12995100
I0920 15:31:36.018773    6388 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:36.018773    6388 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="4b6745cb-38f9-11ed-8eb0-00155d1e80c6" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2" "latency"=7876290400
I0920 15:31:36.019277    6388 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4b6745cb-38f9-11ed-8eb0-00155d1e80c6" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7876290400
I0920 15:31:36.021288    6388 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:36.021770    6388 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="4b67322e-38f9-11ed-8eb0-00155d1e80c6" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b" "latency"=7879788800
I0920 15:31:36.021770    6388 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4b67322e-38f9-11ed-8eb0-00155d1e80c6" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7879788800
I0920 15:31:36.024794    6388 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:31:36.024794    6388 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.azure.com/node-name"="1417k8s00000000" "disk.csi.azure.com/request-id"="4b67322e-38f9-11ed-8eaf-00155d1e80c6" "disk.csi.azure.com/requested-role"="Primary" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/watcher.(*ConditionWaiter).Wait" "disk.csi.azure.com/volume-name"="pvc-2b876417-4866-4173-a2b6-9a5f2335716c" "latency"=7882813100
I0920 15:31:36.024794    6388 crdprovisioner.go:740]  "msg"="Workflow completed with success." "disk.csi.azure.com/request-id"="4b67322e-38f9-11ed-8eaf-00155d1e80c6" "disk.csi.azure.com/requester-name"="sigs.k8s.io/azuredisk-csi-driver/pkg/provisioner.(*CrdProvisioner).WaitForAttach" "latency"=7882813100
I0920 15:31:42.029773    6388 nodeserver_v2.go:211] NodeStageVolume: formatting 6 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-2b876417-4866-4173-a2b6-9a5f2335716c\globalmount with mount options([])
I0920 15:31:42.046270    6388 nodeserver_v2.go:211] NodeStageVolume: formatting 7 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-87fd3abb-9f17-44e3-9da9-ed4523de37d2\globalmount with mount options([])
I0920 15:31:42.272289    6388 nodeserver_v2.go:211] NodeStageVolume: formatting 8 and mounting at \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-8c0db88a-9c46-45e1-9e20-b5ef4253865b\globalmount with mount options([])
I0920 15:31:54.456292    6388 reflector.go:382] sigs.k8s.io/azuredisk-csi-driver/pkg/apis/client/informers/externalversions/factory.go:117: forcing resync
... skipping 848 lines ...
I0920 15:44:03.194609    6388 utils.go:78] GRPC call: /csi.v1.Node/NodeGetCapabilities
I0920 15:44:03.194609    6388 utils.go:79] GRPC request: {}
I0920 15:44:03.194609    6388 utils.go:85] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}}]}
I0920 15:44:03.196575    6388 utils.go:78] GRPC call: /csi.v1.Node/NodeStageVolume
I0920 15:44:03.196575    6388 utils.go:79] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pvc-95fd211e-2f45-48b0-a867-934b746e5775\\globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"btrfs","mount_flags":["barrier","acl"]}},"access_mode":{"mode":7}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-95fd211e-2f45-48b0-a867-934b746e5775","csi.storage.k8s.io/pvc/name":"pvc-mdwt2","csi.storage.k8s.io/pvc/namespace":"azuredisk-9353","fsType":"btrfs","networkAccessPolicy":"AllowAll","requestedsizegib":"1000","skuname":"StandardSSD_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1663687142976-8081-disk.csi.azure.com","zoned":"true"},"volume_id":"/subscriptions/0e46bd28-a80f-4d3a-8200-d9eb8d80cb2e/resourceGroups/kubetest-y9dto3pe/providers/Microsoft.Compute/disks/pvc-95fd211e-2f45-48b0-a867-934b746e5775"}
I0920 15:44:03.196575    6388 conditionwatcher.go:113] Adding a condition function for azvolumeattachments (pvc-95fd211e-2f45-48b0-a867-934b746e5775-1417k8s00000000-attachment)
I0920 15:44:09.780701    6388 conditionwatcher.go:187] condition result: succeeded: true, error: <nil>
I0920 15:44:09.781221    6388 conditionwaiter.go:50]  "msg"="Workflow completed with success." "disk.csi.az