kubernetes-sigs/aws-ebs-csi-driver PR #1233 pull-aws-ebs-csi-driver-external-test #1524178112592482304
Recent runs || View in Spyglass
Recent runs || View in Spyglass
| PR | torredil: Use minimal base image for linux builds |
| Result | ABORTED |
| Tests | 0 failed / 0 succeeded |
| Started | |
| Elapsed | 1h28m |
| Revision | a1ab90890e0e26c98677e8b529abdd7d8fb20f0d |
| Refs |
1233 |
... skipping 588 lines ... ## Validating cluster test-cluster-27998.k8s.local # Using cluster from kubectl context: test-cluster-27998.k8s.local Validating cluster test-cluster-27998.k8s.local W0511 00:04:59.174114 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:05:09.205711 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:05:19.227407 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:05:29.262282 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:05:39.293542 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:05:49.327311 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:05:59.377377 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:06:09.414259 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:06:19.460653 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:06:30.657616 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host W0511 00:06:53.361729 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: an error on the server ("") has prevented the request from succeeding (get nodes) W0511 00:07:14.943523 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: an error on the server ("") has prevented the request from succeeding (get nodes) W0511 00:07:36.525369 9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: an error on the server ("") has prevented the request from succeeding (get nodes) INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master t3.medium 1 1 us-west-2a nodes-us-west-2a Node c5.large 1 1 us-west-2a nodes-us-west-2b Node c5.large 1 1 us-west-2b nodes-us-west-2c Node c5.large 1 1 us-west-2c ... skipping 9 lines ... Machine i-06679b4f8ef30e496 machine "i-06679b4f8ef30e496" has not yet joined cluster Node ip-172-20-52-249.us-west-2.compute.internal node "ip-172-20-52-249.us-west-2.compute.internal" of role "master" is not ready Pod kube-system/coredns-8f5559c9b-qgzpn system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending Pod kube-system/coredns-autoscaler-6f594f4c58-gb99g system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending Pod kube-system/dns-controller-5d59c585d8-nfvwz system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending Validation Failed W0511 00:07:50.496247 9196 validate_cluster.go:221] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master t3.medium 1 1 us-west-2a nodes-us-west-2a Node c5.large 1 1 us-west-2a nodes-us-west-2b Node c5.large 1 1 us-west-2b ... skipping 9 lines ... Machine i-057238a8395634bb6 machine "i-057238a8395634bb6" has not yet joined cluster Machine i-06679b4f8ef30e496 machine "i-06679b4f8ef30e496" has not yet joined cluster Pod kube-system/coredns-8f5559c9b-qgzpn system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending Pod kube-system/coredns-autoscaler-6f594f4c58-gb99g system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending Pod kube-system/dns-controller-5d59c585d8-nfvwz system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending Validation Failed W0511 00:08:02.674657 9196 validate_cluster.go:221] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master t3.medium 1 1 us-west-2a nodes-us-west-2a Node c5.large 1 1 us-west-2a nodes-us-west-2b Node c5.large 1 1 us-west-2b ... skipping 10 lines ... Machine i-06679b4f8ef30e496 machine "i-06679b4f8ef30e496" has not yet joined cluster Pod kube-system/coredns-8f5559c9b-qgzpn system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending Pod kube-system/coredns-autoscaler-6f594f4c58-gb99g system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending Pod kube-system/dns-controller-5d59c585d8-nfvwz system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending Pod kube-system/etcd-manager-main-ip-172-20-52-249.us-west-2.compute.internal system-cluster-critical pod "etcd-manager-main-ip-172-20-52-249.us-west-2.compute.internal" is pending Validation Failed W0511 00:08:14.942216 9196 validate_cluster.go:221] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master t3.medium 1 1 us-west-2a nodes-us-west-2a Node c5.large 1 1 us-west-2a nodes-us-west-2b Node c5.large 1 1 us-west-2b ... skipping 9 lines ... Machine i-057238a8395634bb6 machine "i-057238a8395634bb6" has not yet joined cluster Machine i-06679b4f8ef30e496 machine "i-06679b4f8ef30e496" has not yet joined cluster Pod kube-system/coredns-8f5559c9b-qgzpn system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending Pod kube-system/coredns-autoscaler-6f594f4c58-gb99g system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending Pod kube-system/dns-controller-5d59c585d8-nfvwz system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending Validation Failed W0511 00:08:27.069508 9196 validate_cluster.go:221] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master t3.medium 1 1 us-west-2a nodes-us-west-2a Node c5.large 1 1 us-west-2a nodes-us-west-2b Node c5.large 1 1 us-west-2b ... skipping 11 lines ... Node ip-172-20-52-249.us-west-2.compute.internal master "ip-172-20-52-249.us-west-2.compute.internal" is missing kube-apiserver pod Pod kube-system/coredns-8f5559c9b-qgzpn system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending Pod kube-system/coredns-autoscaler-6f594f4c58-gb99g system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending Pod kube-system/dns-controller-5d59c585d8-nfvwz system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending Pod kube-system/kops-controller-k2pcb system-node-critical pod "kops-controller-k2pcb" is pending Validation Failed W0511 00:08:39.261442 9196 validate_cluster.go:221] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master t3.medium 1 1 us-west-2a nodes-us-west-2a Node c5.large 1 1 us-west-2a nodes-us-west-2b Node c5.large 1 1 us-west-2b ... skipping 8 lines ... Machine i-042bf187445039118 machine "i-042bf187445039118" has not yet joined cluster Machine i-057238a8395634bb6 machine "i-057238a8395634bb6" has not yet joined cluster Machine i-06679b4f8ef30e496 machine "i-06679b4f8ef30e496" has not yet joined cluster Pod kube-system/coredns-8f5559c9b-qgzpn system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending Pod kube-system/coredns-autoscaler-6f594f4c58-gb99g system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending Validation Failed W0511 00:08:51.438174 9196 validate_cluster.go:221] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master t3.medium 1 1 us-west-2a nodes-us-west-2a Node c5.large 1 1 us-west-2a nodes-us-west-2b Node c5.large 1 1 us-west-2b ... skipping 12 lines ... Node ip-172-20-58-208.us-west-2.compute.internal node "ip-172-20-58-208.us-west-2.compute.internal" of role "node" is not ready Node ip-172-20-84-177.us-west-2.compute.internal node "ip-172-20-84-177.us-west-2.compute.internal" of role "node" is not ready Pod kube-system/coredns-8f5559c9b-qgzpn system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending Pod kube-system/coredns-autoscaler-6f594f4c58-gb99g system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending Pod kube-system/kube-proxy-ip-172-20-84-177.us-west-2.compute.internal system-node-critical pod "kube-proxy-ip-172-20-84-177.us-west-2.compute.internal" is pending Validation Failed W0511 00:09:03.991432 9196 validate_cluster.go:221] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master t3.medium 1 1 us-west-2a nodes-us-west-2a Node c5.large 1 1 us-west-2a nodes-us-west-2b Node c5.large 1 1 us-west-2b ... skipping 8 lines ... VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/coredns-8f5559c9b-htw9q system-cluster-critical pod "coredns-8f5559c9b-htw9q" is not ready (coredns) Pod kube-system/coredns-8f5559c9b-qgzpn system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is not ready (coredns) Validation Failed W0511 00:09:16.417888 9196 validate_cluster.go:221] (will retry): cluster not yet healthy INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-us-west-2a Master t3.medium 1 1 us-west-2a nodes-us-west-2a Node c5.large 1 1 us-west-2a nodes-us-west-2b Node c5.large 1 1 us-west-2b ... skipping 196 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:278[0m [36mDriver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m ... skipping 101 lines ... May 11 00:11:07.093: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-514-e2e-scxhzv5 [1mSTEP[0m: creating a claim May 11 00:11:07.161: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-rqfj [1mSTEP[0m: Creating a pod to test atomic-volume-subpath May 11 00:11:07.373: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-rqfj" in namespace "provisioning-514" to be "Succeeded or Failed" May 11 00:11:07.435: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 62.196096ms May 11 00:11:09.499: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.125518858s May 11 00:11:11.562: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.188253688s May 11 00:11:13.625: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.252125131s May 11 00:11:15.689: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.31547365s May 11 00:11:17.753: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.379992196s ... skipping 134 lines ... May 11 00:15:56.365: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.992107235s May 11 00:15:58.428: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.055160938s May 11 00:16:00.492: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.118662638s May 11 00:16:02.557: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.184176036s May 11 00:16:04.620: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.247179641s May 11 00:16:06.683: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.310167183s May 11 00:16:08.830: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-rqfj" container "init-volume-dynamicpv-rqfj": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-rqfj) May 11 00:16:08.894: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-rqfj" container "test-container-subpath-dynamicpv-rqfj": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-rqfj) [1mSTEP[0m: delete the pod May 11 00:16:08.961: INFO: Waiting for pod pod-subpath-test-dynamicpv-rqfj to disappear May 11 00:16:09.024: INFO: Pod pod-subpath-test-dynamicpv-rqfj still exists May 11 00:16:11.024: INFO: Waiting for pod pod-subpath-test-dynamicpv-rqfj to disappear May 11 00:16:11.087: INFO: Pod pod-subpath-test-dynamicpv-rqfj still exists May 11 00:16:13.024: INFO: Waiting for pod pod-subpath-test-dynamicpv-rqfj to disappear ... skipping 25 lines ... May 11 00:16:54.985: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.com6mjcs: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:16:54.985: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.com6mjcs: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-514/ebs.csi.aws.com6mjcs" May 11 00:16:54.985: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.com6mjcs: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:16:54.985: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.com6mjcs: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-6eb1468d-e1cd-4c0c-9c70-4aeee22b6e50 May 11 00:16:54.985: INFO: At 2022-05-11 00:11:14 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {default-scheduler } Scheduled: Successfully assigned provisioning-514/pod-subpath-test-dynamicpv-rqfj to ip-172-20-58-208.us-west-2.compute.internal May 11 00:16:54.985: INFO: At 2022-05-11 00:11:16 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-6eb1468d-e1cd-4c0c-9c70-4aeee22b6e50" May 11 00:16:54.985: INFO: At 2022-05-11 00:11:23 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-6eb1468d-e1cd-4c0c-9c70-4aeee22b6e50" : rpc error: code = Internal desc = could not format "/dev/nvme2n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb1468d-e1cd-4c0c-9c70-4aeee22b6e50/globalmount": failed to get disk format of disk /dev/nvme2n1: signal: segmentation fault (core dumped) May 11 00:16:54.985: INFO: At 2022-05-11 00:13:17 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[default-token-w5mfq test-volume liveness-probe-volume]: timed out waiting for the condition May 11 00:16:54.985: INFO: At 2022-05-11 00:15:35 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-w5mfq]: timed out waiting for the condition May 11 00:16:55.047: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:16:55.047: INFO: May 11 00:16:55.171: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal ... skipping 101 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould support file as subpath [LinuxOnly] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:230[0m [91mUnexpected error: <*errors.errorString | 0xc000e7b1a0>: { s: "expected pod \"pod-subpath-test-dynamicpv-rqfj\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-rqfj\" to be \"Succeeded or Failed\"", } expected pod "pod-subpath-test-dynamicpv-rqfj" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-rqfj" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742 [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow] /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 ... skipping 57 lines ... May 11 00:16:55.587: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comhjlvp: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:16:55.587: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comhjlvp: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "multivolume-304/ebs.csi.aws.comhjlvp" May 11 00:16:55.587: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comhjlvp: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:16:55.587: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.comhjlvp: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-30190e47-d1f3-483a-ad66-ba7af8e91c5a May 11 00:16:55.587: INFO: At 2022-05-11 00:11:14 +0000 UTC - event for pod-85fbad13-d472-44e2-ae52-6d9762809056: {default-scheduler } Scheduled: Successfully assigned multivolume-304/pod-85fbad13-d472-44e2-ae52-6d9762809056 to ip-172-20-58-208.us-west-2.compute.internal May 11 00:16:55.587: INFO: At 2022-05-11 00:11:16 +0000 UTC - event for pod-85fbad13-d472-44e2-ae52-6d9762809056: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-30190e47-d1f3-483a-ad66-ba7af8e91c5a" May 11 00:16:55.587: INFO: At 2022-05-11 00:11:23 +0000 UTC - event for pod-85fbad13-d472-44e2-ae52-6d9762809056: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-30190e47-d1f3-483a-ad66-ba7af8e91c5a" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-30190e47-d1f3-483a-ad66-ba7af8e91c5a/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped) May 11 00:16:55.587: INFO: At 2022-05-11 00:13:17 +0000 UTC - event for pod-85fbad13-d472-44e2-ae52-6d9762809056: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-hht6n]: timed out waiting for the condition May 11 00:16:55.650: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:16:55.650: INFO: May 11 00:16:55.773: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:16:55.836: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 2032 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:14:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:14:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:14:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:14:56 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 100 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould concurrently access the single read-only volume from pods on the same node [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:337[0m [91mUnexpected error: <*errors.errorString | 0xc000ebcd00>: { s: "pod \"pod-85fbad13-d472-44e2-ae52-6d9762809056\" is not Running: timed out waiting for the condition", } pod "pod-85fbad13-d472-44e2-ae52-6d9762809056" is not Running: timed out waiting for the condition occurred[0m ... skipping 92 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Inline-volume (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240[0m [36mDriver "ebs.csi.aws.com" does not support volume type "InlineVolume" - skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m ... skipping 134 lines ... May 11 00:16:59.973: INFO: At 2022-05-11 00:11:08 +0000 UTC - event for ebs.csi.aws.coms4bcd: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:16:59.973: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.com4vnzb: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970 May 11 00:16:59.973: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.coms4bcd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-4c803fa3-fdb4-4375-9b02-f901bc788715 May 11 00:16:59.973: INFO: At 2022-05-11 00:11:14 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {default-scheduler } Scheduled: Successfully assigned multivolume-8585/pod-7fa7162b-0a8d-488f-a29e-29e409853fe8 to ip-172-20-58-208.us-west-2.compute.internal May 11 00:16:59.973: INFO: At 2022-05-11 00:11:16 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-4c803fa3-fdb4-4375-9b02-f901bc788715" May 11 00:16:59.973: INFO: At 2022-05-11 00:11:18 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970" May 11 00:16:59.973: INFO: At 2022-05-11 00:11:19 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-4c803fa3-fdb4-4375-9b02-f901bc788715" : rpc error: code = Internal desc = Could not mount "/dev/nvme3n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-4c803fa3-fdb4-4375-9b02-f901bc788715/423f46a6-d11e-4748-9d4f-bad967b58d15": mount failed: exec: "mount": executable file not found in $PATH Mounting command: mount Mounting arguments: -o bind /dev/nvme3n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-4c803fa3-fdb4-4375-9b02-f901bc788715/423f46a6-d11e-4748-9d4f-bad967b58d15 Output: May 11 00:16:59.973: INFO: At 2022-05-11 00:11:20 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970" : rpc error: code = Internal desc = Could not mount "/dev/nvme4n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970/423f46a6-d11e-4748-9d4f-bad967b58d15": mount failed: exec: "mount": executable file not found in $PATH Mounting command: mount Mounting arguments: -o bind /dev/nvme4n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970/423f46a6-d11e-4748-9d4f-bad967b58d15 Output: May 11 00:16:59.973: INFO: At 2022-05-11 00:13:17 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[default-token-gzl5g volume1 volume2]: timed out waiting for the condition May 11 00:17:00.033: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:17:00.033: INFO: ... skipping 103 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould access to two volumes with the same volume mode and retain data across pod recreation on the same node [LinuxOnly] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:134[0m [91mUnexpected error: <*errors.errorString | 0xc00095ce20>: { s: "pod \"pod-7fa7162b-0a8d-488f-a29e-29e409853fe8\" is not Running: timed out waiting for the condition", } pod "pod-7fa7162b-0a8d-488f-a29e-29e409853fe8" is not Running: timed out waiting for the condition occurred[0m ... skipping 83 lines ... May 11 00:22:45.570: INFO: At 2022-05-11 00:16:59 +0000 UTC - event for ebs.csi.aws.comn2r9v: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:22:45.570: INFO: At 2022-05-11 00:16:59 +0000 UTC - event for ebs.csi.aws.comn2r9v: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "multivolume-6237/ebs.csi.aws.comn2r9v" May 11 00:22:45.570: INFO: At 2022-05-11 00:16:59 +0000 UTC - event for ebs.csi.aws.comn2r9v: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:22:45.570: INFO: At 2022-05-11 00:17:02 +0000 UTC - event for ebs.csi.aws.comn2r9v: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-72673e89-5407-41fa-a39b-a191c254a6ba May 11 00:22:45.570: INFO: At 2022-05-11 00:17:03 +0000 UTC - event for pod-34cc14a3-72d5-4310-9333-6e3a91b006ba: {default-scheduler } Scheduled: Successfully assigned multivolume-6237/pod-34cc14a3-72d5-4310-9333-6e3a91b006ba to ip-172-20-58-208.us-west-2.compute.internal May 11 00:22:45.570: INFO: At 2022-05-11 00:17:05 +0000 UTC - event for pod-34cc14a3-72d5-4310-9333-6e3a91b006ba: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-72673e89-5407-41fa-a39b-a191c254a6ba" May 11 00:22:45.570: INFO: At 2022-05-11 00:17:11 +0000 UTC - event for pod-34cc14a3-72d5-4310-9333-6e3a91b006ba: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-72673e89-5407-41fa-a39b-a191c254a6ba" : rpc error: code = Internal desc = Could not mount "/dev/nvme2n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-72673e89-5407-41fa-a39b-a191c254a6ba/247e98b0-7d5d-4103-949d-df4037c39504": mount failed: exec: "mount": executable file not found in $PATH Mounting command: mount Mounting arguments: -o bind /dev/nvme2n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-72673e89-5407-41fa-a39b-a191c254a6ba/247e98b0-7d5d-4103-949d-df4037c39504 Output: May 11 00:22:45.570: INFO: At 2022-05-11 00:19:06 +0000 UTC - event for pod-34cc14a3-72d5-4310-9333-6e3a91b006ba: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-d5259 volume1]: timed out waiting for the condition May 11 00:22:45.629: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:22:45.629: INFO: ... skipping 103 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould concurrently access the single volume from pods on the same node [LinuxOnly] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:312[0m [91mUnexpected error: <*errors.errorString | 0xc00106eb20>: { s: "pod \"pod-34cc14a3-72d5-4310-9333-6e3a91b006ba\" is not Running: timed out waiting for the condition", } pod "pod-34cc14a3-72d5-4310-9333-6e3a91b006ba" is not Running: timed out waiting for the condition occurred[0m ... skipping 38 lines ... May 11 00:16:57.643: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-3005-e2e-sct78gp [1mSTEP[0m: creating a claim May 11 00:16:57.706: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-zs9l [1mSTEP[0m: Creating a pod to test subpath May 11 00:16:57.895: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-zs9l" in namespace "provisioning-3005" to be "Succeeded or Failed" May 11 00:16:57.958: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 62.178339ms May 11 00:17:00.021: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 2.12544992s May 11 00:17:02.084: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4.188246897s May 11 00:17:04.147: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 6.251271279s May 11 00:17:06.210: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 8.314578544s May 11 00:17:08.273: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 10.377556447s ... skipping 134 lines ... May 11 00:21:46.938: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.042591454s May 11 00:21:49.001: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.105554944s May 11 00:21:51.065: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.169301106s May 11 00:21:53.141: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.24577554s May 11 00:21:55.205: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.309075882s May 11 00:21:57.268: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.372099604s May 11 00:21:59.412: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-zs9l" container "init-volume-dynamicpv-zs9l": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-zs9l) May 11 00:21:59.476: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-zs9l" container "test-init-subpath-dynamicpv-zs9l": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-zs9l) May 11 00:21:59.540: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-zs9l" container "test-container-subpath-dynamicpv-zs9l": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-zs9l) [1mSTEP[0m: delete the pod May 11 00:21:59.604: INFO: Waiting for pod pod-subpath-test-dynamicpv-zs9l to disappear May 11 00:21:59.667: INFO: Pod pod-subpath-test-dynamicpv-zs9l still exists May 11 00:22:01.667: INFO: Waiting for pod pod-subpath-test-dynamicpv-zs9l to disappear May 11 00:22:01.735: INFO: Pod pod-subpath-test-dynamicpv-zs9l still exists May 11 00:22:03.669: INFO: Waiting for pod pod-subpath-test-dynamicpv-zs9l to disappear ... skipping 25 lines ... May 11 00:22:45.619: INFO: At 2022-05-11 00:16:57 +0000 UTC - event for ebs.csi.aws.coms8h4n: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:22:45.619: INFO: At 2022-05-11 00:16:57 +0000 UTC - event for ebs.csi.aws.coms8h4n: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-3005/ebs.csi.aws.coms8h4n" May 11 00:22:45.619: INFO: At 2022-05-11 00:16:57 +0000 UTC - event for ebs.csi.aws.coms8h4n: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:22:45.619: INFO: At 2022-05-11 00:17:01 +0000 UTC - event for ebs.csi.aws.coms8h4n: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-04c19764-71c4-4b24-be43-4a1315e17e34 May 11 00:22:45.619: INFO: At 2022-05-11 00:17:01 +0000 UTC - event for pod-subpath-test-dynamicpv-zs9l: {default-scheduler } Scheduled: Successfully assigned provisioning-3005/pod-subpath-test-dynamicpv-zs9l to ip-172-20-58-208.us-west-2.compute.internal May 11 00:22:45.619: INFO: At 2022-05-11 00:17:03 +0000 UTC - event for pod-subpath-test-dynamicpv-zs9l: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-04c19764-71c4-4b24-be43-4a1315e17e34" May 11 00:22:45.619: INFO: At 2022-05-11 00:17:09 +0000 UTC - event for pod-subpath-test-dynamicpv-zs9l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-04c19764-71c4-4b24-be43-4a1315e17e34" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-04c19764-71c4-4b24-be43-4a1315e17e34/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped) May 11 00:22:45.619: INFO: At 2022-05-11 00:19:04 +0000 UTC - event for pod-subpath-test-dynamicpv-zs9l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-89v5h]: timed out waiting for the condition May 11 00:22:45.689: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:22:45.689: INFO: May 11 00:22:45.753: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:22:45.816: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 3192 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:19:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:19:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:19:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:19:56 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 100 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould support existing directories when readOnly specified in the volumeSource [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:399[0m [91mUnexpected error: <*errors.errorString | 0xc000e7a6f0>: { s: "expected pod \"pod-subpath-test-dynamicpv-zs9l\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-zs9l\" to be \"Succeeded or Failed\"", } expected pod "pod-subpath-test-dynamicpv-zs9l" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-zs9l" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742 [90m------------------------------[0m [36mS[0m [90m------------------------------[0m ... skipping 35 lines ... /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:121 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: ebs.csi.aws.com][0m [90m[Testpattern: Dynamic PV (delayed binding)] topology[0m [1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies[0m [37m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192[0m [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185 [1mSTEP[0m: Creating a kubernetes client May 11 00:22:47.905: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig [1mSTEP[0m: Building a namespace api object, basename topology [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192 May 11 00:22:48.265: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2c] May 11 00:22:48.265: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2a] May 11 00:22:48.266: INFO: Creating storage class object and pvc object for driver - sc: &StorageClass{ObjectMeta:{topology-5303-e2e-scs5479 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{},ReclaimPolicy:nil,MountOptions:[],AllowVolumeExpansion:nil,VolumeBindingMode:*WaitForFirstConsumer,AllowedTopologies:[]TopologySelectorTerm{{[{topology.ebs.csi.aws.com/zone [us-west-2a]}]},},}, pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- topology-5303 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*topology-5303-e2e-scs5479,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} [1mSTEP[0m: Creating sc [1mSTEP[0m: Creating pvc ... skipping 69 lines ... May 11 00:22:50.453: INFO: At 2022-05-11 00:17:02 +0000 UTC - event for ebs.csi.aws.commlpwc: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:22:50.453: INFO: At 2022-05-11 00:17:06 +0000 UTC - event for ebs.csi.aws.com5zcnv: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-83c349f4-0ebe-49c7-ab2b-f77cedfb8371 May 11 00:22:50.453: INFO: At 2022-05-11 00:17:06 +0000 UTC - event for ebs.csi.aws.commlpwc: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-10002f2b-9ab8-43e3-add1-235c361b22ba May 11 00:22:50.453: INFO: At 2022-05-11 00:17:06 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {default-scheduler } Scheduled: Successfully assigned multivolume-5490/pod-81676b9b-64a2-4e2a-be69-fe929301d43d to ip-172-20-58-208.us-west-2.compute.internal May 11 00:22:50.453: INFO: At 2022-05-11 00:17:08 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-10002f2b-9ab8-43e3-add1-235c361b22ba" May 11 00:22:50.453: INFO: At 2022-05-11 00:17:08 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-83c349f4-0ebe-49c7-ab2b-f77cedfb8371" May 11 00:22:50.453: INFO: At 2022-05-11 00:17:10 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-10002f2b-9ab8-43e3-add1-235c361b22ba" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-10002f2b-9ab8-43e3-add1-235c361b22ba/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped) May 11 00:22:50.453: INFO: At 2022-05-11 00:17:10 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-83c349f4-0ebe-49c7-ab2b-f77cedfb8371" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-83c349f4-0ebe-49c7-ab2b-f77cedfb8371/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped) May 11 00:22:50.453: INFO: At 2022-05-11 00:19:09 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume2 volume1], unattached volumes=[volume2 default-token-fvrwj volume1]: timed out waiting for the condition May 11 00:22:50.453: INFO: At 2022-05-11 00:21:25 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[volume1 volume2 default-token-fvrwj]: timed out waiting for the condition May 11 00:22:50.513: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:22:50.513: INFO: May 11 00:22:50.573: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal ... skipping 101 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould access to two volumes with the same volume mode and retain data across pod recreation on the same node [LinuxOnly] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:134[0m [91mUnexpected error: <*errors.errorString | 0xc000f10a70>: { s: "pod \"pod-81676b9b-64a2-4e2a-be69-fe929301d43d\" is not Running: timed out waiting for the condition", } pod "pod-81676b9b-64a2-4e2a-be69-fe929301d43d" is not Running: timed out waiting for the condition occurred[0m ... skipping 16 lines ... May 11 00:11:07.617: INFO: Creating resource for dynamic PV May 11 00:11:07.617: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass snapshotting-8875-e2e-scr87p2 [1mSTEP[0m: creating a claim May 11 00:11:07.682: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: [init] starting a pod to use the claim May 11 00:11:07.877: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-4hxlq" in namespace "snapshotting-8875" to be "Succeeded or Failed" May 11 00:11:07.940: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 63.589663ms May 11 00:11:10.004: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.127461557s May 11 00:11:12.068: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.191260691s May 11 00:11:14.132: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.255068183s May 11 00:11:16.197: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.320105573s May 11 00:11:18.262: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.384943045s ... skipping 424 lines ... May 11 00:25:55.989: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m48.112249496s May 11 00:25:58.056: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m50.17925948s May 11 00:26:00.124: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m52.247344554s May 11 00:26:02.192: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m54.315108137s May 11 00:26:04.259: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m56.382144645s May 11 00:26:06.327: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m58.449662569s May 11 00:26:08.402: INFO: Error getting logs for pod pvc-snapshottable-tester-4hxlq: the server rejected our request for an unknown reason (get pods pvc-snapshottable-tester-4hxlq) May 11 00:26:08.402: INFO: Deleting pod "pvc-snapshottable-tester-4hxlq" in namespace "snapshotting-8875" May 11 00:26:08.471: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-4hxlq" to be fully deleted [AfterEach] volume snapshot controller /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:223 [1mSTEP[0m: Deleting pvc May 11 00:26:18.771: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comb4jmx" ... skipping 14 lines ... May 11 00:26:54.532: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comb4jmx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:26:54.532: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comb4jmx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "snapshotting-8875/ebs.csi.aws.comb4jmx" May 11 00:26:54.532: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comb4jmx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:26:54.532: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.comb4jmx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-568e9664-9e2e-4c06-b5d0-a90058448d83 May 11 00:26:54.532: INFO: At 2022-05-11 00:11:14 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {default-scheduler } Scheduled: Successfully assigned snapshotting-8875/pvc-snapshottable-tester-4hxlq to ip-172-20-58-208.us-west-2.compute.internal May 11 00:26:54.532: INFO: At 2022-05-11 00:11:18 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-568e9664-9e2e-4c06-b5d0-a90058448d83" May 11 00:26:54.532: INFO: At 2022-05-11 00:11:18 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-568e9664-9e2e-4c06-b5d0-a90058448d83" : rpc error: code = Internal desc = could not format "/dev/nvme5n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-568e9664-9e2e-4c06-b5d0-a90058448d83/globalmount": failed to get disk format of disk /dev/nvme5n1: signal: segmentation fault (core dumped) May 11 00:26:54.532: INFO: At 2022-05-11 00:13:17 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[my-volume default-token-6mq28]: timed out waiting for the condition May 11 00:26:54.532: INFO: At 2022-05-11 00:15:34 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[default-token-6mq28 my-volume]: timed out waiting for the condition May 11 00:26:54.598: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:26:54.598: INFO: May 11 00:26:54.665: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal ... skipping 110 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108[0m [91m[1m [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:227[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:245[0m [91mUnexpected error: <*errors.errorString | 0xc000854720>: { s: "Gave up after waiting 15m0s for pod \"pvc-snapshottable-tester-4hxlq\" to be \"Succeeded or Failed\"", } Gave up after waiting 15m0s for pod "pvc-snapshottable-tester-4hxlq" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:675 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m ... skipping 65 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240[0m [36mDriver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m [0mExternal Storage [Driver: ebs.csi.aws.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly][0m [37m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185 [1mSTEP[0m: Creating a kubernetes client May 11 00:22:52.503: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig [1mSTEP[0m: Building a namespace api object, basename provisioning [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [It] should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267 May 11 00:22:52.799: INFO: Creating resource for dynamic PV May 11 00:22:52.800: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-2924-e2e-scrj6k6 [1mSTEP[0m: creating a claim May 11 00:22:52.861: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-cvn4 [1mSTEP[0m: Checking for subpath error in container status May 11 00:27:53.237: INFO: Deleting pod "pod-subpath-test-dynamicpv-cvn4" in namespace "provisioning-2924" May 11 00:27:53.297: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-cvn4" to be fully deleted [1mSTEP[0m: Deleting pod May 11 00:27:59.418: INFO: Deleting pod "pod-subpath-test-dynamicpv-cvn4" in namespace "provisioning-2924" [1mSTEP[0m: Deleting pvc May 11 00:27:59.599: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comnfncb" ... skipping 9 lines ... May 11 00:28:09.967: INFO: At 2022-05-11 00:22:52 +0000 UTC - event for ebs.csi.aws.comnfncb: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:28:09.967: INFO: At 2022-05-11 00:22:53 +0000 UTC - event for ebs.csi.aws.comnfncb: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-2924/ebs.csi.aws.comnfncb" May 11 00:28:09.967: INFO: At 2022-05-11 00:22:53 +0000 UTC - event for ebs.csi.aws.comnfncb: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:28:09.967: INFO: At 2022-05-11 00:22:56 +0000 UTC - event for ebs.csi.aws.comnfncb: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-b2f2a27d-535d-4769-8b8a-321c841e051b May 11 00:28:09.967: INFO: At 2022-05-11 00:22:57 +0000 UTC - event for pod-subpath-test-dynamicpv-cvn4: {default-scheduler } Scheduled: Successfully assigned provisioning-2924/pod-subpath-test-dynamicpv-cvn4 to ip-172-20-58-208.us-west-2.compute.internal May 11 00:28:09.967: INFO: At 2022-05-11 00:22:58 +0000 UTC - event for pod-subpath-test-dynamicpv-cvn4: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-b2f2a27d-535d-4769-8b8a-321c841e051b" May 11 00:28:09.967: INFO: At 2022-05-11 00:23:01 +0000 UTC - event for pod-subpath-test-dynamicpv-cvn4: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-b2f2a27d-535d-4769-8b8a-321c841e051b" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b2f2a27d-535d-4769-8b8a-321c841e051b/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped) May 11 00:28:09.967: INFO: At 2022-05-11 00:25:00 +0000 UTC - event for pod-subpath-test-dynamicpv-cvn4: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-xd2zw]: timed out waiting for the condition May 11 00:28:10.027: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:28:10.027: INFO: May 11 00:28:10.089: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:28:10.149: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 4337 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 101 lines ... [91m[1m• Failure [319.523 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267[0m [91mwhile waiting for subpath failure Unexpected error: <*errors.errorString | 0xc00110fa80>: { s: "error waiting for pod subpath error to occur: timed out waiting for the condition", } error waiting for pod subpath error to occur: timed out waiting for the condition occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:718 [90m------------------------------[0m [0mExternal Storage [Driver: ebs.csi.aws.com][0m [90m[Testpattern: Dynamic PV (default fs)] volumes[0m [1mshould store data[0m ... skipping 25 lines ... May 11 00:27:54.811: INFO: Waiting for pod external-injector to disappear May 11 00:27:54.874: INFO: Pod external-injector still exists May 11 00:27:56.812: INFO: Waiting for pod external-injector to disappear May 11 00:27:56.875: INFO: Pod external-injector still exists May 11 00:27:58.811: INFO: Waiting for pod external-injector to disappear May 11 00:27:58.874: INFO: Pod external-injector no longer exists May 11 00:27:58.874: FAIL: Failed to create injector pod: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3() /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 +0x485 github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes.TestEBSCSI(0x4082b9?) /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes/e2e_test.go:82 +0x311 ... skipping 17 lines ... May 11 00:28:14.454: INFO: At 2022-05-11 00:22:48 +0000 UTC - event for ebs.csi.aws.com6p55l: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:28:14.454: INFO: At 2022-05-11 00:22:48 +0000 UTC - event for ebs.csi.aws.com6p55l: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-1461/ebs.csi.aws.com6p55l" May 11 00:28:14.454: INFO: At 2022-05-11 00:22:48 +0000 UTC - event for ebs.csi.aws.com6p55l: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:28:14.454: INFO: At 2022-05-11 00:22:51 +0000 UTC - event for ebs.csi.aws.com6p55l: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-5485ee07-2692-42b8-a43c-499219b1905f May 11 00:28:14.454: INFO: At 2022-05-11 00:22:52 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned volume-1461/external-injector to ip-172-20-58-208.us-west-2.compute.internal May 11 00:28:14.454: INFO: At 2022-05-11 00:22:56 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-5485ee07-2692-42b8-a43c-499219b1905f" May 11 00:28:14.454: INFO: At 2022-05-11 00:23:00 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-5485ee07-2692-42b8-a43c-499219b1905f" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5485ee07-2692-42b8-a43c-499219b1905f/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped) May 11 00:28:14.454: INFO: At 2022-05-11 00:24:55 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 default-token-47rxx]: timed out waiting for the condition May 11 00:28:14.516: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:28:14.516: INFO: May 11 00:28:14.579: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:28:14.642: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 4337 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 104 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] volumes [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould store data [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159[0m [91mMay 11 00:27:58.874: Failed to create injector pod: timed out waiting for the condition[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: ebs.csi.aws.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m ... skipping 14 lines ... May 11 00:26:57.205: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-4055-e2e-sckg8q7 [1mSTEP[0m: creating a claim May 11 00:26:57.274: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-gbfd [1mSTEP[0m: Creating a pod to test subpath May 11 00:26:57.479: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-gbfd" in namespace "provisioning-4055" to be "Succeeded or Failed" May 11 00:26:57.546: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 66.366694ms May 11 00:26:59.612: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.132754904s May 11 00:27:01.681: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.2011741s May 11 00:27:03.749: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.269021154s May 11 00:27:05.815: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 8.335909992s May 11 00:27:07.884: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 10.404169837s ... skipping 134 lines ... May 11 00:31:47.121: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.641371636s May 11 00:31:49.188: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.708844019s May 11 00:31:51.257: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.777208704s May 11 00:31:53.325: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.845142615s May 11 00:31:55.393: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.913922838s May 11 00:31:57.461: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.981766397s May 11 00:31:59.603: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-gbfd" container "init-volume-dynamicpv-gbfd": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-gbfd) May 11 00:31:59.671: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-gbfd" container "test-init-volume-dynamicpv-gbfd": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-gbfd) May 11 00:31:59.740: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-gbfd" container "test-container-subpath-dynamicpv-gbfd": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-gbfd) [1mSTEP[0m: delete the pod May 11 00:31:59.810: INFO: Waiting for pod pod-subpath-test-dynamicpv-gbfd to disappear May 11 00:31:59.876: INFO: Pod pod-subpath-test-dynamicpv-gbfd still exists May 11 00:32:01.877: INFO: Waiting for pod pod-subpath-test-dynamicpv-gbfd to disappear May 11 00:32:01.947: INFO: Pod pod-subpath-test-dynamicpv-gbfd still exists May 11 00:32:03.877: INFO: Waiting for pod pod-subpath-test-dynamicpv-gbfd to disappear ... skipping 20 lines ... May 11 00:32:20.552: INFO: At 2022-05-11 00:26:57 +0000 UTC - event for ebs.csi.aws.comh2hs2: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:32:20.552: INFO: At 2022-05-11 00:26:57 +0000 UTC - event for ebs.csi.aws.comh2hs2: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-4055/ebs.csi.aws.comh2hs2" May 11 00:32:20.552: INFO: At 2022-05-11 00:26:57 +0000 UTC - event for ebs.csi.aws.comh2hs2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:32:20.552: INFO: At 2022-05-11 00:27:00 +0000 UTC - event for ebs.csi.aws.comh2hs2: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-16a887ee-fa4b-4064-80f5-2fca7b362d44 May 11 00:32:20.552: INFO: At 2022-05-11 00:27:01 +0000 UTC - event for pod-subpath-test-dynamicpv-gbfd: {default-scheduler } Scheduled: Successfully assigned provisioning-4055/pod-subpath-test-dynamicpv-gbfd to ip-172-20-58-208.us-west-2.compute.internal May 11 00:32:20.552: INFO: At 2022-05-11 00:27:03 +0000 UTC - event for pod-subpath-test-dynamicpv-gbfd: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-16a887ee-fa4b-4064-80f5-2fca7b362d44" May 11 00:32:20.552: INFO: At 2022-05-11 00:27:09 +0000 UTC - event for pod-subpath-test-dynamicpv-gbfd: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-16a887ee-fa4b-4064-80f5-2fca7b362d44" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-16a887ee-fa4b-4064-80f5-2fca7b362d44/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped) May 11 00:32:20.552: INFO: At 2022-05-11 00:29:04 +0000 UTC - event for pod-subpath-test-dynamicpv-gbfd: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-lj678]: timed out waiting for the condition May 11 00:32:20.618: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:32:20.618: INFO: May 11 00:32:20.728: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:32:20.794: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 5462 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:29:59 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:29:59 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:29:59 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:29:59 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 104 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould support readOnly directory specified in the volumeMount [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:369[0m [91mUnexpected error: <*errors.errorString | 0xc0011712c0>: { s: "expected pod \"pod-subpath-test-dynamicpv-gbfd\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-gbfd\" to be \"Succeeded or Failed\"", } expected pod "pod-subpath-test-dynamicpv-gbfd" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-gbfd" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m ... skipping 64 lines ... May 11 00:33:13.059: INFO: At 2022-05-11 00:28:12 +0000 UTC - event for pvc-cf2w7: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:33:13.059: INFO: At 2022-05-11 00:28:12 +0000 UTC - event for pvc-cf2w7: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:33:13.059: INFO: At 2022-05-11 00:28:12 +0000 UTC - event for pvc-cf2w7: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-862/pvc-cf2w7" May 11 00:33:13.059: INFO: At 2022-05-11 00:28:16 +0000 UTC - event for pod-6428baf1-8a97-42e3-8ebd-23873519a292: {default-scheduler } Scheduled: Successfully assigned provisioning-862/pod-6428baf1-8a97-42e3-8ebd-23873519a292 to ip-172-20-58-208.us-west-2.compute.internal May 11 00:33:13.059: INFO: At 2022-05-11 00:28:16 +0000 UTC - event for pvc-cf2w7: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-5f93625e-3db8-4ee1-b539-812d61179c45 May 11 00:33:13.059: INFO: At 2022-05-11 00:28:18 +0000 UTC - event for pod-6428baf1-8a97-42e3-8ebd-23873519a292: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-5f93625e-3db8-4ee1-b539-812d61179c45" May 11 00:33:13.059: INFO: At 2022-05-11 00:28:20 +0000 UTC - event for pod-6428baf1-8a97-42e3-8ebd-23873519a292: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-5f93625e-3db8-4ee1-b539-812d61179c45" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5f93625e-3db8-4ee1-b539-812d61179c45/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped) May 11 00:33:13.059: INFO: At 2022-05-11 00:30:19 +0000 UTC - event for pod-6428baf1-8a97-42e3-8ebd-23873519a292: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-k4zc7]: timed out waiting for the condition May 11 00:33:13.119: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:33:13.119: INFO: pod-6428baf1-8a97-42e3-8ebd-23873519a292 ip-172-20-58-208.us-west-2.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:16 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:16 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:16 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:16 +0000 UTC }] May 11 00:33:13.119: INFO: May 11 00:33:13.180: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal ... skipping 107 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] provisioning [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould provision storage with mount options [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:179[0m [91mUnexpected error: <*errors.errorString | 0xc00095cc00>: { s: "pod \"pod-6428baf1-8a97-42e3-8ebd-23873519a292\" is not Running: timed out waiting for the condition", } pod "pod-6428baf1-8a97-42e3-8ebd-23873519a292" is not Running: timed out waiting for the condition occurred[0m ... skipping 39 lines ... May 11 00:37:45.528: INFO: At 2022-05-11 00:32:23 +0000 UTC - event for ebs.csi.aws.comz58f6: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:37:45.528: INFO: At 2022-05-11 00:32:23 +0000 UTC - event for ebs.csi.aws.comz58f6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-1782/ebs.csi.aws.comz58f6" May 11 00:37:45.528: INFO: At 2022-05-11 00:32:23 +0000 UTC - event for ebs.csi.aws.comz58f6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:37:45.528: INFO: At 2022-05-11 00:32:26 +0000 UTC - event for ebs.csi.aws.comz58f6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-dd22cb69-e292-4c49-b739-695dfa3de75d May 11 00:37:45.528: INFO: At 2022-05-11 00:32:27 +0000 UTC - event for pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2: {default-scheduler } Scheduled: Successfully assigned volume-expand-1782/pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2 to ip-172-20-58-208.us-west-2.compute.internal May 11 00:37:45.528: INFO: At 2022-05-11 00:32:29 +0000 UTC - event for pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-dd22cb69-e292-4c49-b739-695dfa3de75d" May 11 00:37:45.528: INFO: At 2022-05-11 00:32:31 +0000 UTC - event for pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-dd22cb69-e292-4c49-b739-695dfa3de75d" : rpc error: code = Internal desc = Could not mount "/dev/nvme4n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-dd22cb69-e292-4c49-b739-695dfa3de75d/e3aa5ee0-d83b-4179-8c4b-0c2b2ca18ce7": mount failed: exec: "mount": executable file not found in $PATH Mounting command: mount Mounting arguments: -o bind /dev/nvme4n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-dd22cb69-e292-4c49-b739-695dfa3de75d/e3aa5ee0-d83b-4179-8c4b-0c2b2ca18ce7 Output: May 11 00:37:45.528: INFO: At 2022-05-11 00:34:30 +0000 UTC - event for pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-bz859 volume1]: timed out waiting for the condition May 11 00:37:45.597: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:37:45.597: INFO: ... skipping 108 lines ... [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mVerify if offline PVC expansion works [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174[0m [91mWhile creating pods for resizing Unexpected error: <*errors.errorString | 0xc000d26090>: { s: "pod \"pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2\" is not Running: timed out waiting for the condition", } pod "pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2" is not Running: timed out waiting for the condition occurred[0m ... skipping 100 lines ... May 11 00:22:49.258: INFO: Creating resource for dynamic PV May 11 00:22:49.258: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass snapshotting-745-e2e-sckvt92 [1mSTEP[0m: creating a claim May 11 00:22:49.319: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: [init] starting a pod to use the claim May 11 00:22:49.505: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-rvm4s" in namespace "snapshotting-745" to be "Succeeded or Failed" May 11 00:22:49.565: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 60.029248ms May 11 00:22:51.635: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 2.129770944s May 11 00:22:53.696: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4.191065408s May 11 00:22:55.758: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 6.252870772s May 11 00:22:57.819: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 8.314460168s May 11 00:22:59.881: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 10.375897268s ... skipping 425 lines ... May 11 00:37:38.809: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m49.303929217s May 11 00:37:40.873: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m51.368271716s May 11 00:37:42.938: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m53.432538161s May 11 00:37:45.002: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m55.496678874s May 11 00:37:47.066: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m57.561307372s May 11 00:37:49.136: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m59.631200364s May 11 00:37:51.201: INFO: Error getting logs for pod pvc-snapshottable-tester-rvm4s: the server rejected our request for an unknown reason (get pods pvc-snapshottable-tester-rvm4s) May 11 00:37:51.201: INFO: Deleting pod "pvc-snapshottable-tester-rvm4s" in namespace "snapshotting-745" May 11 00:37:51.265: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-rvm4s" to be fully deleted [AfterEach] volume snapshot controller /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:223 [1mSTEP[0m: Deleting pvc May 11 00:37:59.516: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comlzpmt" ... skipping 9 lines ... May 11 00:38:09.905: INFO: At 2022-05-11 00:22:49 +0000 UTC - event for ebs.csi.aws.comlzpmt: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:38:09.905: INFO: At 2022-05-11 00:22:49 +0000 UTC - event for ebs.csi.aws.comlzpmt: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "snapshotting-745/ebs.csi.aws.comlzpmt" May 11 00:38:09.905: INFO: At 2022-05-11 00:22:49 +0000 UTC - event for ebs.csi.aws.comlzpmt: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:38:09.905: INFO: At 2022-05-11 00:22:52 +0000 UTC - event for ebs.csi.aws.comlzpmt: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-f4383d4f-4ef3-48bb-9906-67bbb78e61a2 May 11 00:38:09.905: INFO: At 2022-05-11 00:22:53 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {default-scheduler } Scheduled: Successfully assigned snapshotting-745/pvc-snapshottable-tester-rvm4s to ip-172-20-58-208.us-west-2.compute.internal May 11 00:38:09.905: INFO: At 2022-05-11 00:22:57 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f4383d4f-4ef3-48bb-9906-67bbb78e61a2" May 11 00:38:09.905: INFO: At 2022-05-11 00:23:01 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-f4383d4f-4ef3-48bb-9906-67bbb78e61a2" : rpc error: code = Internal desc = could not format "/dev/nvme2n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f4383d4f-4ef3-48bb-9906-67bbb78e61a2/globalmount": failed to get disk format of disk /dev/nvme2n1: signal: segmentation fault (core dumped) May 11 00:38:09.905: INFO: At 2022-05-11 00:24:56 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[my-volume default-token-9lzt8]: timed out waiting for the condition May 11 00:38:09.905: INFO: At 2022-05-11 00:36:20 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[default-token-9lzt8 my-volume]: timed out waiting for the condition May 11 00:38:09.968: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:38:09.968: INFO: May 11 00:38:10.033: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal ... skipping 109 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108[0m [91m[1m [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:227[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:245[0m [91mUnexpected error: <*errors.errorString | 0xc000ec0500>: { s: "Gave up after waiting 15m0s for pod \"pvc-snapshottable-tester-rvm4s\" to be \"Succeeded or Failed\"", } Gave up after waiting 15m0s for pod "pvc-snapshottable-tester-rvm4s" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:675 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m ... skipping 14 lines ... May 11 00:28:16.879: INFO: Creating resource for dynamic PV May 11 00:28:16.879: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-7648-e2e-scnwzwz [1mSTEP[0m: creating a claim May 11 00:28:16.942: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod to format volume volume-prep-provisioning-7648 May 11 00:28:17.136: INFO: Waiting up to 5m0s for pod "volume-prep-provisioning-7648" in namespace "provisioning-7648" to be "Succeeded or Failed" May 11 00:28:17.198: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 62.290659ms May 11 00:28:19.261: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 2.125545261s May 11 00:28:21.325: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 4.189537195s May 11 00:28:23.389: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 6.253355519s May 11 00:28:25.452: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 8.316860034s May 11 00:28:27.516: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 10.380834754s ... skipping 210 lines ... May 11 00:38:22.750: INFO: At 2022-05-11 00:28:17 +0000 UTC - event for ebs.csi.aws.coml86g9: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:38:22.750: INFO: At 2022-05-11 00:28:17 +0000 UTC - event for ebs.csi.aws.coml86g9: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-7648/ebs.csi.aws.coml86g9" May 11 00:38:22.750: INFO: At 2022-05-11 00:28:17 +0000 UTC - event for ebs.csi.aws.coml86g9: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:38:22.750: INFO: At 2022-05-11 00:28:20 +0000 UTC - event for ebs.csi.aws.coml86g9: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-84a89b77-3de7-41cf-aa28-e12be2c5d992 May 11 00:38:22.750: INFO: At 2022-05-11 00:28:21 +0000 UTC - event for volume-prep-provisioning-7648: {default-scheduler } Scheduled: Successfully assigned provisioning-7648/volume-prep-provisioning-7648 to ip-172-20-58-208.us-west-2.compute.internal May 11 00:38:22.750: INFO: At 2022-05-11 00:28:23 +0000 UTC - event for volume-prep-provisioning-7648: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-84a89b77-3de7-41cf-aa28-e12be2c5d992" May 11 00:38:22.750: INFO: At 2022-05-11 00:28:37 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-84a89b77-3de7-41cf-aa28-e12be2c5d992" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-84a89b77-3de7-41cf-aa28-e12be2c5d992/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped) May 11 00:38:22.750: INFO: At 2022-05-11 00:30:24 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume default-token-82s79]: timed out waiting for the condition May 11 00:38:22.750: INFO: At 2022-05-11 00:32:42 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[default-token-82s79 test-volume]: timed out waiting for the condition May 11 00:38:22.751: INFO: At 2022-05-11 00:35:13 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[default-token-82s79 test-volume]: error processing PVC provisioning-7648/ebs.csi.aws.coml86g9: PVC is being deleted May 11 00:38:22.751: INFO: At 2022-05-11 00:35:25 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume default-token-82s79]: error processing PVC provisioning-7648/ebs.csi.aws.coml86g9: PVC is being deleted May 11 00:38:22.812: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:38:22.812: INFO: volume-prep-provisioning-7648 ip-172-20-58-208.us-west-2.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:21 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:21 +0000 UTC ContainersNotReady containers with unready status: [init-volume-provisioning-7648]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:21 +0000 UTC ContainersNotReady containers with unready status: [init-volume-provisioning-7648]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:21 +0000 UTC }] May 11 00:38:22.812: INFO: May 11 00:38:22.875: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:38:22.997: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 6539 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 107 lines ... [Testpattern: Dynamic PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould verify container cannot write to subpath readonly volumes [Slow] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:427[0m [91mwhile waiting for volume init pod to succeed Unexpected error: <*errors.errorString | 0xc000e7b970>: { s: "Gave up after waiting 5m0s for pod \"volume-prep-provisioning-7648\" to be \"Succeeded or Failed\"", } Gave up after waiting 5m0s for pod "volume-prep-provisioning-7648" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:1021 [90m------------------------------[0m [36mS[0m [90m------------------------------[0m ... skipping 15 lines ... May 11 00:33:15.421: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass volume-718-e2e-sc9fjj4 [1mSTEP[0m: creating a claim May 11 00:33:15.482: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-zknn [1mSTEP[0m: Creating a pod to test exec-volume-test May 11 00:33:15.667: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-zknn" in namespace "volume-718" to be "Succeeded or Failed" May 11 00:33:15.726: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 59.449473ms May 11 00:33:17.787: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.120179768s May 11 00:33:19.848: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.181218643s May 11 00:33:21.909: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.242141011s May 11 00:33:23.970: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.303205829s May 11 00:33:26.032: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.365303911s ... skipping 134 lines ... May 11 00:38:04.294: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.627324704s May 11 00:38:06.356: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.689019173s May 11 00:38:08.417: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.750081875s May 11 00:38:10.478: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.810943904s May 11 00:38:12.538: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.870734141s May 11 00:38:14.599: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.93234705s May 11 00:38:16.772: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "exec-volume-test-dynamicpv-zknn" container "exec-container-dynamicpv-zknn": the server rejected our request for an unknown reason (get pods exec-volume-test-dynamicpv-zknn) [1mSTEP[0m: delete the pod May 11 00:38:16.834: INFO: Waiting for pod exec-volume-test-dynamicpv-zknn to disappear May 11 00:38:16.903: INFO: Pod exec-volume-test-dynamicpv-zknn still exists May 11 00:38:18.903: INFO: Waiting for pod exec-volume-test-dynamicpv-zknn to disappear May 11 00:38:18.964: INFO: Pod exec-volume-test-dynamicpv-zknn still exists May 11 00:38:20.903: INFO: Waiting for pod exec-volume-test-dynamicpv-zknn to disappear ... skipping 21 lines ... May 11 00:38:44.511: INFO: At 2022-05-11 00:33:15 +0000 UTC - event for ebs.csi.aws.com4p82g: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:38:44.511: INFO: At 2022-05-11 00:33:15 +0000 UTC - event for ebs.csi.aws.com4p82g: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-718/ebs.csi.aws.com4p82g" May 11 00:38:44.511: INFO: At 2022-05-11 00:33:15 +0000 UTC - event for ebs.csi.aws.com4p82g: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:38:44.511: INFO: At 2022-05-11 00:33:18 +0000 UTC - event for ebs.csi.aws.com4p82g: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-f0894353-965d-4327-8e02-4fdafbce2a94 May 11 00:38:44.511: INFO: At 2022-05-11 00:33:19 +0000 UTC - event for exec-volume-test-dynamicpv-zknn: {default-scheduler } Scheduled: Successfully assigned volume-718/exec-volume-test-dynamicpv-zknn to ip-172-20-58-208.us-west-2.compute.internal May 11 00:38:44.511: INFO: At 2022-05-11 00:33:21 +0000 UTC - event for exec-volume-test-dynamicpv-zknn: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f0894353-965d-4327-8e02-4fdafbce2a94" May 11 00:38:44.511: INFO: At 2022-05-11 00:33:23 +0000 UTC - event for exec-volume-test-dynamicpv-zknn: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-f0894353-965d-4327-8e02-4fdafbce2a94" : rpc error: code = Internal desc = could not format "/dev/nvme5n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f0894353-965d-4327-8e02-4fdafbce2a94/globalmount": failed to get disk format of disk /dev/nvme5n1: signal: segmentation fault (core dumped) May 11 00:38:44.511: INFO: At 2022-05-11 00:35:22 +0000 UTC - event for exec-volume-test-dynamicpv-zknn: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[vol1], unattached volumes=[default-token-c7xvv vol1]: timed out waiting for the condition May 11 00:38:44.570: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:38:44.570: INFO: May 11 00:38:44.630: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:38:44.690: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 6539 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 104 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (xfs)][Slow] volumes [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould allow exec of files on the volume [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:196[0m [91mUnexpected error: <*errors.errorString | 0xc0009bc730>: { s: "expected pod \"exec-volume-test-dynamicpv-zknn\" success: Gave up after waiting 5m0s for pod \"exec-volume-test-dynamicpv-zknn\" to be \"Succeeded or Failed\"", } expected pod "exec-volume-test-dynamicpv-zknn" success: Gave up after waiting 5m0s for pod "exec-volume-test-dynamicpv-zknn" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742 [90m------------------------------[0m [36mS[0m [90m------------------------------[0m ... skipping 56 lines ... May 11 00:42:54.839: INFO: Waiting for pod external-injector to disappear May 11 00:42:54.907: INFO: Pod external-injector still exists May 11 00:42:56.840: INFO: Waiting for pod external-injector to disappear May 11 00:42:56.907: INFO: Pod external-injector still exists May 11 00:42:58.839: INFO: Waiting for pod external-injector to disappear May 11 00:42:58.906: INFO: Pod external-injector no longer exists May 11 00:42:58.906: FAIL: Failed to create injector pod: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.prepareSnapshotDataSourceForProvisioning(_, {{0xc000668588, 0x11}, {0x281b564, 0x8}, {0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...) /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:845 +0x2e5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*provisioningTestSuite).DefineTests.func4() /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:219 +0x565 ... skipping 10 lines ... May 11 00:42:58.974: INFO: At 2022-05-11 00:37:48 +0000 UTC - event for pvc-7jv5w: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:42:58.974: INFO: At 2022-05-11 00:37:48 +0000 UTC - event for pvc-7jv5w: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:42:58.974: INFO: At 2022-05-11 00:37:48 +0000 UTC - event for pvc-7jv5w: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-9780/pvc-7jv5w" May 11 00:42:58.974: INFO: At 2022-05-11 00:37:51 +0000 UTC - event for pvc-7jv5w: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326 May 11 00:42:58.974: INFO: At 2022-05-11 00:37:52 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned provisioning-9780/external-injector to ip-172-20-58-208.us-west-2.compute.internal May 11 00:42:58.974: INFO: At 2022-05-11 00:37:54 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326" May 11 00:42:58.974: INFO: At 2022-05-11 00:37:56 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326" : rpc error: code = Internal desc = Could not mount "/dev/nvme1n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326/2f3a4d6d-b61b-4760-b803-4c161a00c99e": mount failed: exec: "mount": executable file not found in $PATH Mounting command: mount Mounting arguments: -o bind /dev/nvme1n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326/2f3a4d6d-b61b-4760-b803-4c161a00c99e Output: May 11 00:42:58.974: INFO: At 2022-05-11 00:39:55 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[default-token-g5btb external-volume-0]: timed out waiting for the condition May 11 00:42:59.041: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:42:59.041: INFO: ... skipping 108 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (block volmode)] provisioning [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:200[0m [91mMay 11 00:42:58.906: Failed to create injector pod: timed out waiting for the condition[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:845 [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 May 11 00:43:01.225: INFO: Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping ... skipping 137 lines ... May 11 00:43:34.667: INFO: At 2022-05-11 00:38:12 +0000 UTC - event for ebs.csi.aws.com9xp9m: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:43:34.667: INFO: At 2022-05-11 00:38:12 +0000 UTC - event for ebs.csi.aws.com9xp9m: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-449/ebs.csi.aws.com9xp9m" May 11 00:43:34.667: INFO: At 2022-05-11 00:38:12 +0000 UTC - event for ebs.csi.aws.com9xp9m: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:43:34.668: INFO: At 2022-05-11 00:38:15 +0000 UTC - event for ebs.csi.aws.com9xp9m: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5 May 11 00:43:34.668: INFO: At 2022-05-11 00:38:16 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {default-scheduler } Scheduled: Successfully assigned volume-expand-449/pod-988c7646-bdd1-4d20-8724-748b64f2ac31 to ip-172-20-58-208.us-west-2.compute.internal May 11 00:43:34.668: INFO: At 2022-05-11 00:38:18 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5" May 11 00:43:34.668: INFO: At 2022-05-11 00:38:24 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5" : rpc error: code = Internal desc = Could not mount "/dev/nvme2n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5/7f2a9b3e-2c9f-4cf4-86ac-f9ae4d40d19c": mount failed: exec: "mount": executable file not found in $PATH Mounting command: mount Mounting arguments: -o bind /dev/nvme2n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5/7f2a9b3e-2c9f-4cf4-86ac-f9ae4d40d19c Output: May 11 00:43:34.668: INFO: At 2022-05-11 00:40:19 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-bqs8c]: timed out waiting for the condition May 11 00:43:34.668: INFO: At 2022-05-11 00:42:35 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-bqs8c volume1]: timed out waiting for the condition May 11 00:43:34.731: INFO: POD NODE PHASE GRACE CONDITIONS ... skipping 110 lines ... [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould resize volume when PVC is edited while pod is using it [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246[0m [91mWhile creating pods for resizing Unexpected error: <*errors.errorString | 0xc000e84850>: { s: "pod \"pod-988c7646-bdd1-4d20-8724-748b64f2ac31\" is not Running: timed out waiting for the condition", } pod "pod-988c7646-bdd1-4d20-8724-748b64f2ac31" is not Running: timed out waiting for the condition occurred[0m ... skipping 77 lines ... /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m [36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: ebs.csi.aws.com][0m [90m[Testpattern: Dynamic PV (default fs)] subPath[0m [1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly][0m [37m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240[0m [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185 [1mSTEP[0m: Creating a kubernetes client May 11 00:38:46.971: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig [1mSTEP[0m: Building a namespace api object, basename provisioning [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [It] should fail if subpath directory is outside the volume [Slow][LinuxOnly] /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240 May 11 00:38:47.270: INFO: Creating resource for dynamic PV May 11 00:38:47.270: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-6601-e2e-sc4rktk [1mSTEP[0m: creating a claim May 11 00:38:47.330: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-hc2l [1mSTEP[0m: Checking for subpath error in container status May 11 00:43:47.692: INFO: Deleting pod "pod-subpath-test-dynamicpv-hc2l" in namespace "provisioning-6601" May 11 00:43:47.755: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-hc2l" to be fully deleted [1mSTEP[0m: Deleting pod May 11 00:43:59.877: INFO: Deleting pod "pod-subpath-test-dynamicpv-hc2l" in namespace "provisioning-6601" [1mSTEP[0m: Deleting pvc May 11 00:44:00.057: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comrdr6x" ... skipping 10 lines ... May 11 00:44:15.484: INFO: At 2022-05-11 00:38:47 +0000 UTC - event for ebs.csi.aws.comrdr6x: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:44:15.484: INFO: At 2022-05-11 00:38:47 +0000 UTC - event for ebs.csi.aws.comrdr6x: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-6601/ebs.csi.aws.comrdr6x" May 11 00:44:15.484: INFO: At 2022-05-11 00:38:47 +0000 UTC - event for ebs.csi.aws.comrdr6x: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:44:15.484: INFO: At 2022-05-11 00:38:50 +0000 UTC - event for ebs.csi.aws.comrdr6x: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-a421b3a2-148a-4428-af8c-d4db1354169f May 11 00:44:15.484: INFO: At 2022-05-11 00:38:51 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {default-scheduler } Scheduled: Successfully assigned provisioning-6601/pod-subpath-test-dynamicpv-hc2l to ip-172-20-58-208.us-west-2.compute.internal May 11 00:44:15.484: INFO: At 2022-05-11 00:38:53 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-a421b3a2-148a-4428-af8c-d4db1354169f" May 11 00:44:15.484: INFO: At 2022-05-11 00:38:55 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-a421b3a2-148a-4428-af8c-d4db1354169f" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a421b3a2-148a-4428-af8c-d4db1354169f/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped) May 11 00:44:15.484: INFO: At 2022-05-11 00:40:54 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[default-token-rkmdb test-volume liveness-probe-volume]: timed out waiting for the condition May 11 00:44:15.484: INFO: At 2022-05-11 00:43:10 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-rkmdb]: timed out waiting for the condition May 11 00:44:15.544: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:44:15.544: INFO: May 11 00:44:15.605: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal ... skipping 102 lines ... [91m[1m• Failure [330.577 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240[0m [91mwhile waiting for subpath failure Unexpected error: <*errors.errorString | 0xc000df8690>: { s: "error waiting for pod subpath error to occur: timed out waiting for the condition", } error waiting for pod subpath error to occur: timed out waiting for the condition occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:718 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m ... skipping 73 lines ... May 11 00:48:25.868: INFO: At 2022-05-11 00:43:01 +0000 UTC - event for ebs.csi.aws.comd7qq8: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:48:25.868: INFO: At 2022-05-11 00:43:01 +0000 UTC - event for ebs.csi.aws.comd7qq8: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "multivolume-4341/ebs.csi.aws.comd7qq8" May 11 00:48:25.868: INFO: At 2022-05-11 00:43:01 +0000 UTC - event for ebs.csi.aws.comd7qq8: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:48:25.868: INFO: At 2022-05-11 00:43:05 +0000 UTC - event for ebs.csi.aws.comd7qq8: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-19813fee-5743-4aab-aaca-c7531eb29322 May 11 00:48:25.868: INFO: At 2022-05-11 00:43:05 +0000 UTC - event for pod-10936628-816b-4642-99b9-62fcb80b1a42: {default-scheduler } Scheduled: Successfully assigned multivolume-4341/pod-10936628-816b-4642-99b9-62fcb80b1a42 to ip-172-20-58-208.us-west-2.compute.internal May 11 00:48:25.868: INFO: At 2022-05-11 00:43:09 +0000 UTC - event for pod-10936628-816b-4642-99b9-62fcb80b1a42: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-19813fee-5743-4aab-aaca-c7531eb29322" May 11 00:48:25.868: INFO: At 2022-05-11 00:43:21 +0000 UTC - event for pod-10936628-816b-4642-99b9-62fcb80b1a42: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-19813fee-5743-4aab-aaca-c7531eb29322" : rpc error: code = Internal desc = Could not mount "/dev/nvme1n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-19813fee-5743-4aab-aaca-c7531eb29322/62728771-9321-49bf-97fb-1d037bcc797b": mount failed: exec: "mount": executable file not found in $PATH Mounting command: mount Mounting arguments: -o bind /dev/nvme1n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-19813fee-5743-4aab-aaca-c7531eb29322/62728771-9321-49bf-97fb-1d037bcc797b Output: May 11 00:48:25.868: INFO: At 2022-05-11 00:45:08 +0000 UTC - event for pod-10936628-816b-4642-99b9-62fcb80b1a42: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-l22vh volume1]: timed out waiting for the condition May 11 00:48:25.931: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:48:25.931: INFO: ... skipping 107 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould concurrently access the single read-only volume from pods on the same node [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:337[0m [91mUnexpected error: <*errors.errorString | 0xc000650c00>: { s: "pod \"pod-10936628-816b-4642-99b9-62fcb80b1a42\" is not Running: timed out waiting for the condition", } pod "pod-10936628-816b-4642-99b9-62fcb80b1a42" is not Running: timed out waiting for the condition occurred[0m ... skipping 67 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Pre-provisioned PV (block volmode)] volumeMode [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297[0m [36mDriver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m ... skipping 31 lines ... May 11 00:49:26.419: INFO: Waiting for pod external-injector to disappear May 11 00:49:26.482: INFO: Pod external-injector still exists May 11 00:49:28.421: INFO: Waiting for pod external-injector to disappear May 11 00:49:28.484: INFO: Pod external-injector still exists May 11 00:49:30.419: INFO: Waiting for pod external-injector to disappear May 11 00:49:30.482: INFO: Pod external-injector no longer exists May 11 00:49:30.482: FAIL: Failed to create injector pod: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3() /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 +0x485 github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes.TestEBSCSI(0x0?) /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes/e2e_test.go:82 +0x311 ... skipping 17 lines ... May 11 00:49:46.055: INFO: At 2022-05-11 00:44:17 +0000 UTC - event for ebs.csi.aws.comvbr2r: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:49:46.055: INFO: At 2022-05-11 00:44:18 +0000 UTC - event for ebs.csi.aws.comvbr2r: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-5268/ebs.csi.aws.comvbr2r" May 11 00:49:46.055: INFO: At 2022-05-11 00:44:18 +0000 UTC - event for ebs.csi.aws.comvbr2r: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:49:46.055: INFO: At 2022-05-11 00:44:21 +0000 UTC - event for ebs.csi.aws.comvbr2r: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-75405376-72c7-4c02-ad51-fc2ac9604439 May 11 00:49:46.055: INFO: At 2022-05-11 00:44:22 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned volume-5268/external-injector to ip-172-20-58-208.us-west-2.compute.internal May 11 00:49:46.055: INFO: At 2022-05-11 00:44:24 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-75405376-72c7-4c02-ad51-fc2ac9604439" May 11 00:49:46.055: INFO: At 2022-05-11 00:44:25 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-75405376-72c7-4c02-ad51-fc2ac9604439" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-75405376-72c7-4c02-ad51-fc2ac9604439/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped) May 11 00:49:46.055: INFO: At 2022-05-11 00:46:25 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[default-token-j8p46 external-volume-0]: timed out waiting for the condition May 11 00:49:46.055: INFO: At 2022-05-11 00:48:39 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 default-token-j8p46]: timed out waiting for the condition May 11 00:49:46.117: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:49:46.117: INFO: May 11 00:49:46.180: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal ... skipping 105 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (xfs)][Slow] volumes [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould store data [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159[0m [91mMay 11 00:49:30.482: Failed to create injector pod: timed out waiting for the condition[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Inline-volume (default fs)] subPath ... skipping 127 lines ... May 11 00:53:41.753: INFO: At 2022-05-11 00:43:37 +0000 UTC - event for ebs.csi.aws.comspczh: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:53:41.753: INFO: At 2022-05-11 00:43:37 +0000 UTC - event for ebs.csi.aws.comspczh: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-368/ebs.csi.aws.comspczh" May 11 00:53:41.753: INFO: At 2022-05-11 00:43:37 +0000 UTC - event for ebs.csi.aws.comspczh: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:53:41.753: INFO: At 2022-05-11 00:43:40 +0000 UTC - event for ebs.csi.aws.comspczh: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-91290b1c-6974-4222-9ba7-b492c4852d5c May 11 00:53:41.753: INFO: At 2022-05-11 00:43:41 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-368/pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc to ip-172-20-58-208.us-west-2.compute.internal May 11 00:53:41.753: INFO: At 2022-05-11 00:43:43 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-91290b1c-6974-4222-9ba7-b492c4852d5c" May 11 00:53:41.753: INFO: At 2022-05-11 00:43:45 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-91290b1c-6974-4222-9ba7-b492c4852d5c" : rpc error: code = Internal desc = could not format "/dev/nvme2n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-91290b1c-6974-4222-9ba7-b492c4852d5c/globalmount": failed to get disk format of disk /dev/nvme2n1: signal: segmentation fault (core dumped) May 11 00:53:41.753: INFO: At 2022-05-11 00:45:44 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-25hd6]: timed out waiting for the condition May 11 00:53:41.753: INFO: At 2022-05-11 00:50:30 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-25hd6]: error processing PVC fsgroupchangepolicy-368/ebs.csi.aws.comspczh: PVC is being deleted May 11 00:53:41.815: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:53:41.815: INFO: pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc ip-172-20-58-208.us-west-2.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:43:41 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:43:41 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:43:41 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:43:41 +0000 UTC }] May 11 00:53:41.815: INFO: May 11 00:53:41.877: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:53:41.940: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 9915 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 104 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1m(Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208[0m [91mUnexpected error: <*errors.errorString | 0xc000e859f0>: { s: "pod \"pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc\" is not Running: timed out waiting for the condition", } pod "pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc" is not Running: timed out waiting for the condition occurred[0m ... skipping 129 lines ... /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:121 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: ebs.csi.aws.com][0m [90m[Testpattern: Dynamic PV (block volmode)] volumeMode[0m [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m [37m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297[0m [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185 [1mSTEP[0m: Creating a kubernetes client May 11 00:53:44.385: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig [1mSTEP[0m: Building a namespace api object, basename volumemode [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [It] should fail to use a volume in a pod with mismatched mode [Slow] /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297 May 11 00:53:44.693: INFO: Creating resource for dynamic PV May 11 00:53:44.693: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass volumemode-1189-e2e-sc7gm6c [1mSTEP[0m: creating a claim [1mSTEP[0m: Creating pod [1mSTEP[0m: Waiting for the pod to fail May 11 00:53:51.129: INFO: Deleting pod "pod-73bb5975-8d3a-4977-96c5-df68dbddfaa5" in namespace "volumemode-1189" May 11 00:53:51.193: INFO: Wait up to 5m0s for pod "pod-73bb5975-8d3a-4977-96c5-df68dbddfaa5" to be fully deleted [1mSTEP[0m: Deleting pvc May 11 00:53:59.441: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com6nrgn" May 11 00:53:59.503: INFO: Waiting up to 5m0s for PersistentVolume pvc-23a6249b-ea7b-41b1-bddf-ecd415806f51 to get deleted May 11 00:53:59.564: INFO: PersistentVolume pvc-23a6249b-ea7b-41b1-bddf-ecd415806f51 found and phase=Released (60.899036ms) ... skipping 7 lines ... [32m• [SLOW TEST:20.432 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (block volmode)] volumeMode [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m should fail to use a volume in a pod with mismatched mode [Slow] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297[0m [90m------------------------------[0m [36mS[0m[36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: ebs.csi.aws.com][0m [90m[Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource][0m [0mvolume snapshot controller[0m [90m[0m [1mshould check snapshot fields, check restore correctly works after modifying source data, check deletion[0m ... skipping 12 lines ... May 11 00:38:25.238: INFO: Creating resource for dynamic PV May 11 00:38:25.238: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass snapshotting-7107-e2e-scchsnn [1mSTEP[0m: creating a claim May 11 00:38:25.301: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: [init] starting a pod to use the claim May 11 00:38:25.496: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-dnhvk" in namespace "snapshotting-7107" to be "Succeeded or Failed" May 11 00:38:25.558: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 61.838627ms May 11 00:38:27.621: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 2.124265578s May 11 00:38:29.684: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 4.187645262s May 11 00:38:31.748: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 6.25117358s May 11 00:38:33.810: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 8.313391461s May 11 00:38:35.874: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 10.377360624s ... skipping 424 lines ... May 11 00:53:13.365: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m47.869022719s May 11 00:53:15.431: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m49.934972559s May 11 00:53:17.496: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m51.99973686s May 11 00:53:19.562: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m54.065730536s May 11 00:53:21.629: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m56.1327446s May 11 00:53:23.694: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m58.197557875s May 11 00:53:25.766: INFO: Error getting logs for pod pvc-snapshottable-tester-dnhvk: the server rejected our request for an unknown reason (get pods pvc-snapshottable-tester-dnhvk) May 11 00:53:25.766: INFO: Deleting pod "pvc-snapshottable-tester-dnhvk" in namespace "snapshotting-7107" May 11 00:53:25.832: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-dnhvk" to be fully deleted [AfterEach] volume snapshot controller /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:223 [1mSTEP[0m: Deleting pvc May 11 00:53:30.092: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comhmmcd" ... skipping 14 lines ... May 11 00:54:05.844: INFO: At 2022-05-11 00:38:25 +0000 UTC - event for ebs.csi.aws.comhmmcd: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:54:05.844: INFO: At 2022-05-11 00:38:25 +0000 UTC - event for ebs.csi.aws.comhmmcd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "snapshotting-7107/ebs.csi.aws.comhmmcd" May 11 00:54:05.844: INFO: At 2022-05-11 00:38:25 +0000 UTC - event for ebs.csi.aws.comhmmcd: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:54:05.844: INFO: At 2022-05-11 00:38:28 +0000 UTC - event for ebs.csi.aws.comhmmcd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-41b1404e-2a2b-4bf8-af71-80141d1608f4 May 11 00:54:05.844: INFO: At 2022-05-11 00:38:29 +0000 UTC - event for pvc-snapshottable-tester-dnhvk: {default-scheduler } Scheduled: Successfully assigned snapshotting-7107/pvc-snapshottable-tester-dnhvk to ip-172-20-58-208.us-west-2.compute.internal May 11 00:54:05.844: INFO: At 2022-05-11 00:38:31 +0000 UTC - event for pvc-snapshottable-tester-dnhvk: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-41b1404e-2a2b-4bf8-af71-80141d1608f4" May 11 00:54:05.844: INFO: At 2022-05-11 00:38:33 +0000 UTC - event for pvc-snapshottable-tester-dnhvk: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-41b1404e-2a2b-4bf8-af71-80141d1608f4" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41b1404e-2a2b-4bf8-af71-80141d1608f4/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped) May 11 00:54:05.844: INFO: At 2022-05-11 00:40:32 +0000 UTC - event for pvc-snapshottable-tester-dnhvk: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[my-volume default-token-jjfwc]: timed out waiting for the condition May 11 00:54:05.908: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:54:05.908: INFO: May 11 00:54:05.973: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:54:06.039: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 9915 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 106 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108[0m [91m[1m [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:227[0m should check snapshot fields, check restore correctly works after modifying source data, check deletion [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:245[0m [91mUnexpected error: <*errors.errorString | 0xc000e7a9d0>: { s: "Gave up after waiting 15m0s for pod \"pvc-snapshottable-tester-dnhvk\" to be \"Succeeded or Failed\"", } Gave up after waiting 15m0s for pod "pvc-snapshottable-tester-dnhvk" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:675 [90m------------------------------[0m [36mS[0m [90m------------------------------[0m ... skipping 6 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Inline-volume (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:256[0m [36mDriver "ebs.csi.aws.com" does not support volume type "InlineVolume" - skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m ... skipping 29 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Inline-volume (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:278[0m [36mDriver "ebs.csi.aws.com" does not support volume type "InlineVolume" - skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m ... skipping 64 lines ... /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:372 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m [0mExternal Storage [Driver: ebs.csi.aws.com][0m [90m[Testpattern: Dynamic PV (filesystem volmode)] volumeMode[0m [1mshould fail to use a volume in a pod with mismatched mode [Slow][0m [37m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297[0m [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185 [1mSTEP[0m: Creating a kubernetes client May 11 00:54:08.562: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig [1mSTEP[0m: Building a namespace api object, basename volumemode [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [It] should fail to use a volume in a pod with mismatched mode [Slow] /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297 May 11 00:54:08.887: INFO: Creating resource for dynamic PV May 11 00:54:08.887: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass volumemode-3181-e2e-scmv558 [1mSTEP[0m: creating a claim [1mSTEP[0m: Creating pod [1mSTEP[0m: Waiting for the pod to fail May 11 00:54:15.357: INFO: Deleting pod "pod-4a8173af-146c-41bf-a656-b727c763696b" in namespace "volumemode-3181" May 11 00:54:15.422: INFO: Wait up to 5m0s for pod "pod-4a8173af-146c-41bf-a656-b727c763696b" to be fully deleted [1mSTEP[0m: Deleting pvc May 11 00:54:19.681: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comkx6zt" May 11 00:54:19.747: INFO: Waiting up to 5m0s for PersistentVolume pvc-8f11720b-90e3-43cb-b257-00a5b5358822 to get deleted May 11 00:54:19.811: INFO: PersistentVolume pvc-8f11720b-90e3-43cb-b257-00a5b5358822 found and phase=Released (64.328985ms) ... skipping 7 lines ... [32m• [SLOW TEST:16.512 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (filesystem volmode)] volumeMode [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m should fail to use a volume in a pod with mismatched mode [Slow] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 May 11 00:54:25.075: INFO: Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping [AfterEach] [Testpattern: Pre-provisioned PV (default fs)] subPath ... skipping 2 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:256[0m [36mDriver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m [0mExternal Storage [Driver: ebs.csi.aws.com][0m [90m[Testpattern: Dynamic PV (immediate binding)] topology[0m [1mshould fail to schedule a pod which has topologies that conflict with AllowedTopologies[0m [37m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192[0m [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51 [BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185 [1mSTEP[0m: Creating a kubernetes client May 11 00:54:25.077: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig [1mSTEP[0m: Building a namespace api object, basename topology [1mSTEP[0m: Waiting for a default service account to be provisioned in namespace [It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192 May 11 00:54:25.464: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2c] May 11 00:54:25.465: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2a] May 11 00:54:25.465: INFO: Creating storage class object and pvc object for driver - sc: &StorageClass{ObjectMeta:{topology-8346-e2e-sct7vcr 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{},ReclaimPolicy:nil,MountOptions:[],AllowVolumeExpansion:nil,VolumeBindingMode:*Immediate,AllowedTopologies:[]TopologySelectorTerm{{[{topology.ebs.csi.aws.com/zone [us-west-2c]}]},},}, pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- topology-8346 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*topology-8346-e2e-sct7vcr,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},} [1mSTEP[0m: Creating sc [1mSTEP[0m: Creating pvc ... skipping 148 lines ... May 11 00:55:35.021: INFO: At 2022-05-11 00:49:48 +0000 UTC - event for pvc-jfqn6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:55:35.021: INFO: At 2022-05-11 00:49:48 +0000 UTC - event for pvc-jfqn6: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:55:35.021: INFO: At 2022-05-11 00:49:48 +0000 UTC - event for pvc-jfqn6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "topology-5464/pvc-jfqn6" May 11 00:55:35.021: INFO: At 2022-05-11 00:49:52 +0000 UTC - event for pvc-jfqn6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-cd897812-8156-49c9-bc2a-fe57fd7b6ff1 May 11 00:55:35.021: INFO: At 2022-05-11 00:49:53 +0000 UTC - event for pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed: {default-scheduler } Scheduled: Successfully assigned topology-5464/pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed to ip-172-20-58-208.us-west-2.compute.internal May 11 00:55:35.021: INFO: At 2022-05-11 00:49:55 +0000 UTC - event for pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-cd897812-8156-49c9-bc2a-fe57fd7b6ff1" May 11 00:55:35.021: INFO: At 2022-05-11 00:49:57 +0000 UTC - event for pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-cd897812-8156-49c9-bc2a-fe57fd7b6ff1" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cd897812-8156-49c9-bc2a-fe57fd7b6ff1/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped) May 11 00:55:35.021: INFO: At 2022-05-11 00:51:56 +0000 UTC - event for pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-kknk7]: timed out waiting for the condition May 11 00:55:35.083: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:55:35.083: INFO: May 11 00:55:35.146: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:55:35.208: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 11143 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 104 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (delayed binding)] topology [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould provision a volume and schedule a pod with AllowedTopologies [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:164[0m [91mUnexpected error: <*errors.errorString | 0xc0001b0200>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred[0m ... skipping 90 lines ... May 11 00:58:33.116: INFO: At 2022-05-11 00:48:28 +0000 UTC - event for ebs.csi.aws.comvstp8: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 00:58:33.116: INFO: At 2022-05-11 00:48:28 +0000 UTC - event for ebs.csi.aws.comvstp8: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-3578/ebs.csi.aws.comvstp8" May 11 00:58:33.116: INFO: At 2022-05-11 00:48:28 +0000 UTC - event for ebs.csi.aws.comvstp8: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:58:33.116: INFO: At 2022-05-11 00:48:31 +0000 UTC - event for ebs.csi.aws.comvstp8: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-f896ae32-7a0d-47f3-b1fc-ecdc5f856453 May 11 00:58:33.116: INFO: At 2022-05-11 00:48:32 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-3578/pod-54bcd33a-ebbb-444d-b919-3b04b75461cb to ip-172-20-58-208.us-west-2.compute.internal May 11 00:58:33.116: INFO: At 2022-05-11 00:48:34 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f896ae32-7a0d-47f3-b1fc-ecdc5f856453" May 11 00:58:33.116: INFO: At 2022-05-11 00:48:40 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-f896ae32-7a0d-47f3-b1fc-ecdc5f856453" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f896ae32-7a0d-47f3-b1fc-ecdc5f856453/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped) May 11 00:58:33.116: INFO: At 2022-05-11 00:50:35 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-cv7gc]: timed out waiting for the condition May 11 00:58:33.116: INFO: At 2022-05-11 00:55:21 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-cv7gc]: error processing PVC fsgroupchangepolicy-3578/ebs.csi.aws.comvstp8: PVC is being deleted May 11 00:58:33.116: INFO: At 2022-05-11 00:55:34 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-cv7gc volume1]: error processing PVC fsgroupchangepolicy-3578/ebs.csi.aws.comvstp8: PVC is being deleted May 11 00:58:33.182: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:58:33.182: INFO: pod-54bcd33a-ebbb-444d-b919-3b04b75461cb ip-172-20-58-208.us-west-2.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:48:32 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:48:32 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:48:32 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:48:32 +0000 UTC }] May 11 00:58:33.182: INFO: May 11 00:58:33.250: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 00:58:33.315: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 11143 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 108 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1m(Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208[0m [91mUnexpected error: <*errors.errorString | 0xc000d7cc10>: { s: "pod \"pod-54bcd33a-ebbb-444d-b919-3b04b75461cb\" is not Running: timed out waiting for the condition", } pod "pod-54bcd33a-ebbb-444d-b919-3b04b75461cb" is not Running: timed out waiting for the condition occurred[0m ... skipping 59 lines ... May 11 00:59:20.332: INFO: At 2022-05-11 00:54:05 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {default-scheduler } FailedScheduling: 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims. May 11 00:59:20.332: INFO: At 2022-05-11 00:54:05 +0000 UTC - event for pvc-h7zwd: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 00:59:20.332: INFO: At 2022-05-11 00:54:05 +0000 UTC - event for pvc-h7zwd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "topology-1172/pvc-h7zwd" May 11 00:59:20.332: INFO: At 2022-05-11 00:54:08 +0000 UTC - event for pvc-h7zwd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-b3bb2cad-3ece-433e-af63-5356213b8d90 May 11 00:59:20.332: INFO: At 2022-05-11 00:54:13 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {default-scheduler } Scheduled: Successfully assigned topology-1172/pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d to ip-172-20-58-208.us-west-2.compute.internal May 11 00:59:20.332: INFO: At 2022-05-11 00:54:15 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-b3bb2cad-3ece-433e-af63-5356213b8d90" May 11 00:59:20.332: INFO: At 2022-05-11 00:54:21 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-b3bb2cad-3ece-433e-af63-5356213b8d90" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b3bb2cad-3ece-433e-af63-5356213b8d90/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped) May 11 00:59:20.332: INFO: At 2022-05-11 00:56:16 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-gj8gj]: timed out waiting for the condition May 11 00:59:20.332: INFO: At 2022-05-11 00:58:31 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-gj8gj volume1]: timed out waiting for the condition May 11 00:59:20.396: INFO: POD NODE PHASE GRACE CONDITIONS May 11 00:59:20.396: INFO: May 11 00:59:20.460: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal ... skipping 107 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (immediate binding)] topology [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould provision a volume and schedule a pod with AllowedTopologies [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:164[0m [91mUnexpected error: <*errors.errorString | 0xc00021c1d0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred[0m ... skipping 19 lines ... May 11 00:55:37.484: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-9086-e2e-scdp2cz [1mSTEP[0m: creating a claim May 11 00:55:37.547: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-hcll [1mSTEP[0m: Creating a pod to test subpath May 11 00:55:37.737: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-hcll" in namespace "provisioning-9086" to be "Succeeded or Failed" May 11 00:55:37.799: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 62.124875ms May 11 00:55:39.867: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 2.12960971s May 11 00:55:41.930: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4.192522117s May 11 00:55:43.993: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 6.255893518s May 11 00:55:46.056: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 8.319279374s May 11 00:55:48.120: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 10.3824466s ... skipping 134 lines ... May 11 01:00:26.764: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.02633169s May 11 01:00:28.828: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.090290015s May 11 01:00:30.891: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.153644063s May 11 01:00:32.954: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.216772742s May 11 01:00:35.018: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.280438904s May 11 01:00:37.082: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.344416449s May 11 01:00:39.218: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-hcll" container "init-volume-dynamicpv-hcll": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-hcll) May 11 01:00:39.280: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-hcll" container "test-init-volume-dynamicpv-hcll": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-hcll) May 11 01:00:39.342: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-hcll" container "test-container-subpath-dynamicpv-hcll": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-hcll) [1mSTEP[0m: delete the pod May 11 01:00:39.404: INFO: Waiting for pod pod-subpath-test-dynamicpv-hcll to disappear May 11 01:00:39.464: INFO: Pod pod-subpath-test-dynamicpv-hcll still exists May 11 01:00:41.466: INFO: Waiting for pod pod-subpath-test-dynamicpv-hcll to disappear May 11 01:00:41.527: INFO: Pod pod-subpath-test-dynamicpv-hcll still exists May 11 01:00:43.465: INFO: Waiting for pod pod-subpath-test-dynamicpv-hcll to disappear ... skipping 21 lines ... May 11 01:01:05.146: INFO: At 2022-05-11 00:55:37 +0000 UTC - event for ebs.csi.aws.com2mqtx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 01:01:05.146: INFO: At 2022-05-11 00:55:37 +0000 UTC - event for ebs.csi.aws.com2mqtx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-9086/ebs.csi.aws.com2mqtx" May 11 01:01:05.146: INFO: At 2022-05-11 00:55:37 +0000 UTC - event for ebs.csi.aws.com2mqtx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 01:01:05.146: INFO: At 2022-05-11 00:55:41 +0000 UTC - event for ebs.csi.aws.com2mqtx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-29de2a7d-293f-472b-8f18-15e80949cd22 May 11 01:01:05.146: INFO: At 2022-05-11 00:55:41 +0000 UTC - event for pod-subpath-test-dynamicpv-hcll: {default-scheduler } Scheduled: Successfully assigned provisioning-9086/pod-subpath-test-dynamicpv-hcll to ip-172-20-58-208.us-west-2.compute.internal May 11 01:01:05.146: INFO: At 2022-05-11 00:55:43 +0000 UTC - event for pod-subpath-test-dynamicpv-hcll: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-29de2a7d-293f-472b-8f18-15e80949cd22" May 11 01:01:05.146: INFO: At 2022-05-11 00:55:49 +0000 UTC - event for pod-subpath-test-dynamicpv-hcll: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-29de2a7d-293f-472b-8f18-15e80949cd22" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-29de2a7d-293f-472b-8f18-15e80949cd22/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped) May 11 01:01:05.146: INFO: At 2022-05-11 00:57:44 +0000 UTC - event for pod-subpath-test-dynamicpv-hcll: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-dczjf]: timed out waiting for the condition May 11 01:01:05.206: INFO: POD NODE PHASE GRACE CONDITIONS May 11 01:01:05.206: INFO: May 11 01:01:05.272: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 01:01:05.332: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 12255 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 105 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould support readOnly file specified in the volumeMount [LinuxOnly] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:384[0m [91mUnexpected error: <*errors.errorString | 0xc0007ca6b0>: { s: "expected pod \"pod-subpath-test-dynamicpv-hcll\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-hcll\" to be \"Succeeded or Failed\"", } expected pod "pod-subpath-test-dynamicpv-hcll" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-hcll" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742 [90m------------------------------[0m [36mS[0m [90m------------------------------[0m ... skipping 134 lines ... May 11 01:03:36.269: INFO: Waiting for pod external-injector to disappear May 11 01:03:36.332: INFO: Pod external-injector still exists May 11 01:03:38.333: INFO: Waiting for pod external-injector to disappear May 11 01:03:38.396: INFO: Pod external-injector still exists May 11 01:03:40.332: INFO: Waiting for pod external-injector to disappear May 11 01:03:40.395: INFO: Pod external-injector no longer exists May 11 01:03:40.395: FAIL: Failed to create injector pod: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.prepareSnapshotDataSourceForProvisioning(_, {{0xc000ca45e8, 0x11}, {0x281b564, 0x8}, {0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...) /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:845 +0x2e5 k8s.io/kubernetes/test/e2e/storage/testsuites.(*provisioningTestSuite).DefineTests.func4() /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:219 +0x565 ... skipping 10 lines ... May 11 01:03:40.460: INFO: At 2022-05-11 00:58:35 +0000 UTC - event for pvc-v262f: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 01:03:40.460: INFO: At 2022-05-11 00:58:36 +0000 UTC - event for pvc-v262f: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 01:03:40.460: INFO: At 2022-05-11 00:58:36 +0000 UTC - event for pvc-v262f: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-8211/pvc-v262f" May 11 01:03:40.460: INFO: At 2022-05-11 00:58:39 +0000 UTC - event for pvc-v262f: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-972585f9-1635-4b28-9ebb-eed239d81d95 May 11 01:03:40.460: INFO: At 2022-05-11 00:58:40 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned provisioning-8211/external-injector to ip-172-20-58-208.us-west-2.compute.internal May 11 01:03:40.460: INFO: At 2022-05-11 00:58:42 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-972585f9-1635-4b28-9ebb-eed239d81d95" May 11 01:03:40.460: INFO: At 2022-05-11 00:58:55 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-972585f9-1635-4b28-9ebb-eed239d81d95" : rpc error: code = Internal desc = could not format "/dev/nvme5n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-972585f9-1635-4b28-9ebb-eed239d81d95/globalmount": failed to get disk format of disk /dev/nvme5n1: signal: segmentation fault (core dumped) May 11 01:03:40.460: INFO: At 2022-05-11 01:00:43 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 default-token-wptcn]: timed out waiting for the condition May 11 01:03:40.522: INFO: POD NODE PHASE GRACE CONDITIONS May 11 01:03:40.522: INFO: May 11 01:03:40.586: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 01:03:40.649: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 12255 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 105 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] provisioning [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:200[0m [91mMay 11 01:03:40.395: Failed to create injector pod: timed out waiting for the condition[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:845 [90m------------------------------[0m [36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath ... skipping 150 lines ... May 11 01:04:31.839: INFO: At 2022-05-11 00:54:27 +0000 UTC - event for ebs.csi.aws.com2m7jx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 01:04:31.839: INFO: At 2022-05-11 00:54:27 +0000 UTC - event for ebs.csi.aws.com2m7jx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-5351/ebs.csi.aws.com2m7jx" May 11 01:04:31.839: INFO: At 2022-05-11 00:54:27 +0000 UTC - event for ebs.csi.aws.com2m7jx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 01:04:31.839: INFO: At 2022-05-11 00:54:30 +0000 UTC - event for ebs.csi.aws.com2m7jx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-4c7a1ae8-cc68-4821-b0c5-fcf206df0b14 May 11 01:04:31.839: INFO: At 2022-05-11 00:54:31 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-5351/pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce to ip-172-20-58-208.us-west-2.compute.internal May 11 01:04:31.839: INFO: At 2022-05-11 00:54:33 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-4c7a1ae8-cc68-4821-b0c5-fcf206df0b14" May 11 01:04:31.839: INFO: At 2022-05-11 00:54:39 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-4c7a1ae8-cc68-4821-b0c5-fcf206df0b14" : rpc error: code = Internal desc = could not format "/dev/nvme2n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4c7a1ae8-cc68-4821-b0c5-fcf206df0b14/globalmount": failed to get disk format of disk /dev/nvme2n1: signal: segmentation fault (core dumped) May 11 01:04:31.839: INFO: At 2022-05-11 00:56:34 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-b85sh]: timed out waiting for the condition May 11 01:04:31.839: INFO: At 2022-05-11 01:01:23 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-b85sh]: error processing PVC fsgroupchangepolicy-5351/ebs.csi.aws.com2m7jx: PVC is being deleted May 11 01:04:31.839: INFO: At 2022-05-11 01:03:09 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-b85sh volume1]: error processing PVC fsgroupchangepolicy-5351/ebs.csi.aws.com2m7jx: PVC is being deleted May 11 01:04:31.903: INFO: POD NODE PHASE GRACE CONDITIONS May 11 01:04:31.903: INFO: pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce ip-172-20-58-208.us-west-2.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:54:31 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:54:31 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:54:31 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:54:31 +0000 UTC }] May 11 01:04:31.903: INFO: May 11 01:04:32.019: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 01:04:32.086: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 12255 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 104 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1m(OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208[0m [91mUnexpected error: <*errors.errorString | 0xc000bef4b0>: { s: "pod \"pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce\" is not Running: timed out waiting for the condition", } pod "pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce" is not Running: timed out waiting for the condition occurred[0m ... skipping 19 lines ... May 11 00:59:22.770: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass provisioning-5676-e2e-sc4r66s [1mSTEP[0m: creating a claim May 11 00:59:22.834: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod pod-subpath-test-dynamicpv-8sr8 [1mSTEP[0m: Creating a pod to test subpath May 11 00:59:23.027: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-8sr8" in namespace "provisioning-5676" to be "Succeeded or Failed" May 11 00:59:23.090: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 62.997809ms May 11 00:59:25.154: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.126870771s May 11 00:59:27.217: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.190496891s May 11 00:59:29.281: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.25422595s May 11 00:59:31.345: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.318077861s May 11 00:59:33.409: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.382322582s ... skipping 134 lines ... May 11 01:04:12.192: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.164678583s May 11 01:04:14.255: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.227695792s May 11 01:04:16.319: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.291799386s May 11 01:04:18.383: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.355717571s May 11 01:04:20.446: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.41924009s May 11 01:04:22.509: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.482176849s May 11 01:04:24.639: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-8sr8" container "init-volume-dynamicpv-8sr8": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-8sr8) May 11 01:04:24.711: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-8sr8" container "test-container-subpath-dynamicpv-8sr8": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-8sr8) [1mSTEP[0m: delete the pod May 11 01:04:24.775: INFO: Waiting for pod pod-subpath-test-dynamicpv-8sr8 to disappear May 11 01:04:24.838: INFO: Pod pod-subpath-test-dynamicpv-8sr8 still exists May 11 01:04:26.839: INFO: Waiting for pod pod-subpath-test-dynamicpv-8sr8 to disappear May 11 01:04:26.905: INFO: Pod pod-subpath-test-dynamicpv-8sr8 still exists May 11 01:04:28.838: INFO: Waiting for pod pod-subpath-test-dynamicpv-8sr8 to disappear ... skipping 15 lines ... May 11 01:04:44.546: INFO: At 2022-05-11 00:59:22 +0000 UTC - event for ebs.csi.aws.comqszv6: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 01:04:44.546: INFO: At 2022-05-11 00:59:23 +0000 UTC - event for ebs.csi.aws.comqszv6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-5676/ebs.csi.aws.comqszv6" May 11 01:04:44.546: INFO: At 2022-05-11 00:59:23 +0000 UTC - event for ebs.csi.aws.comqszv6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 01:04:44.546: INFO: At 2022-05-11 00:59:26 +0000 UTC - event for ebs.csi.aws.comqszv6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-08fa6a3f-6bf5-4e29-949d-7b759b03166a May 11 01:04:44.546: INFO: At 2022-05-11 00:59:27 +0000 UTC - event for pod-subpath-test-dynamicpv-8sr8: {default-scheduler } Scheduled: Successfully assigned provisioning-5676/pod-subpath-test-dynamicpv-8sr8 to ip-172-20-58-208.us-west-2.compute.internal May 11 01:04:44.546: INFO: At 2022-05-11 00:59:28 +0000 UTC - event for pod-subpath-test-dynamicpv-8sr8: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-08fa6a3f-6bf5-4e29-949d-7b759b03166a" May 11 01:04:44.546: INFO: At 2022-05-11 00:59:30 +0000 UTC - event for pod-subpath-test-dynamicpv-8sr8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-08fa6a3f-6bf5-4e29-949d-7b759b03166a" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-08fa6a3f-6bf5-4e29-949d-7b759b03166a/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped) May 11 01:04:44.546: INFO: At 2022-05-11 01:01:30 +0000 UTC - event for pod-subpath-test-dynamicpv-8sr8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-ptx8z]: timed out waiting for the condition May 11 01:04:44.609: INFO: POD NODE PHASE GRACE CONDITIONS May 11 01:04:44.609: INFO: May 11 01:04:44.672: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 01:04:44.735: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 12255 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 106 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould support existing single file [LinuxOnly] [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:219[0m [91mUnexpected error: <*errors.errorString | 0xc000ab4f50>: { s: "expected pod \"pod-subpath-test-dynamicpv-8sr8\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-8sr8\" to be \"Succeeded or Failed\"", } expected pod "pod-subpath-test-dynamicpv-8sr8" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-8sr8" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742 [90m------------------------------[0m [36mS[0m[36mS[0m[36mS[0m [90m------------------------------[0m ... skipping 46 lines ... May 11 01:04:34.346: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} [1mSTEP[0m: creating a StorageClass volume-7154-e2e-scrts84 [1mSTEP[0m: creating a claim May 11 01:04:34.411: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil [1mSTEP[0m: Creating pod exec-volume-test-dynamicpv-llsv [1mSTEP[0m: Creating a pod to test exec-volume-test May 11 01:04:34.608: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-llsv" in namespace "volume-7154" to be "Succeeded or Failed" May 11 01:04:34.673: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 64.429394ms May 11 01:04:36.737: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.129275792s May 11 01:04:38.802: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.194094111s May 11 01:04:40.868: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.259542802s May 11 01:04:42.933: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.325115844s May 11 01:04:44.999: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.390464897s ... skipping 134 lines ... May 11 01:09:23.856: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.247705716s May 11 01:09:25.920: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.312227721s May 11 01:09:27.986: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.37754408s May 11 01:09:30.052: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.4435775s May 11 01:09:32.117: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.509306208s May 11 01:09:34.182: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.573550902s May 11 01:09:36.327: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "exec-volume-test-dynamicpv-llsv" container "exec-container-dynamicpv-llsv": the server rejected our request for an unknown reason (get pods exec-volume-test-dynamicpv-llsv) [1mSTEP[0m: delete the pod May 11 01:09:36.394: INFO: Waiting for pod exec-volume-test-dynamicpv-llsv to disappear May 11 01:09:36.459: INFO: Pod exec-volume-test-dynamicpv-llsv still exists May 11 01:09:38.460: INFO: Waiting for pod exec-volume-test-dynamicpv-llsv to disappear May 11 01:09:38.524: INFO: Pod exec-volume-test-dynamicpv-llsv still exists May 11 01:09:40.459: INFO: Waiting for pod exec-volume-test-dynamicpv-llsv to disappear ... skipping 13 lines ... May 11 01:09:56.110: INFO: At 2022-05-11 01:04:34 +0000 UTC - event for ebs.csi.aws.comkq4km: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 01:09:56.110: INFO: At 2022-05-11 01:04:34 +0000 UTC - event for ebs.csi.aws.comkq4km: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-7154/ebs.csi.aws.comkq4km" May 11 01:09:56.110: INFO: At 2022-05-11 01:04:34 +0000 UTC - event for ebs.csi.aws.comkq4km: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 01:09:56.110: INFO: At 2022-05-11 01:04:37 +0000 UTC - event for ebs.csi.aws.comkq4km: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-bc3bdf1d-ea10-43dd-a13f-0704ea409c92 May 11 01:09:56.110: INFO: At 2022-05-11 01:04:38 +0000 UTC - event for exec-volume-test-dynamicpv-llsv: {default-scheduler } Scheduled: Successfully assigned volume-7154/exec-volume-test-dynamicpv-llsv to ip-172-20-58-208.us-west-2.compute.internal May 11 01:09:56.110: INFO: At 2022-05-11 01:04:40 +0000 UTC - event for exec-volume-test-dynamicpv-llsv: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-bc3bdf1d-ea10-43dd-a13f-0704ea409c92" May 11 01:09:56.111: INFO: At 2022-05-11 01:04:42 +0000 UTC - event for exec-volume-test-dynamicpv-llsv: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-bc3bdf1d-ea10-43dd-a13f-0704ea409c92" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bc3bdf1d-ea10-43dd-a13f-0704ea409c92/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped) May 11 01:09:56.111: INFO: At 2022-05-11 01:06:41 +0000 UTC - event for exec-volume-test-dynamicpv-llsv: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[vol1], unattached volumes=[vol1 default-token-2zzh7]: timed out waiting for the condition May 11 01:09:56.174: INFO: POD NODE PHASE GRACE CONDITIONS May 11 01:09:56.174: INFO: May 11 01:09:56.239: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 01:09:56.302: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 13431 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:05:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:05:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:05:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:05:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 104 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] volumes [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould allow exec of files on the volume [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:196[0m [91mUnexpected error: <*errors.errorString | 0xc00062f1e0>: { s: "expected pod \"exec-volume-test-dynamicpv-llsv\" success: Gave up after waiting 5m0s for pod \"exec-volume-test-dynamicpv-llsv\" to be \"Succeeded or Failed\"", } expected pod "exec-volume-test-dynamicpv-llsv" success: Gave up after waiting 5m0s for pod "exec-volume-test-dynamicpv-llsv" to be "Succeeded or Failed" occurred[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742 [90m------------------------------[0m [36mS[0m[36mS[0m [90m------------------------------[0m ... skipping 6 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Pre-provisioned PV (default fs)] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267[0m [36mDriver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m ... skipping 37 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297[0m [36mDriver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263 [90m------------------------------[0m ... skipping 27 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267[0m [36mDriver ebs.csi.aws.com doesn't support ntfs -- skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:121 [90m------------------------------[0m ... skipping 91 lines ... May 11 01:10:15.753: INFO: At 2022-05-11 01:04:47 +0000 UTC - event for ebs.csi.aws.comhm6kv: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 01:10:15.753: INFO: At 2022-05-11 01:04:47 +0000 UTC - event for ebs.csi.aws.comhm6kv: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-4949/ebs.csi.aws.comhm6kv" May 11 01:10:15.753: INFO: At 2022-05-11 01:04:47 +0000 UTC - event for ebs.csi.aws.comhm6kv: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 01:10:15.753: INFO: At 2022-05-11 01:04:51 +0000 UTC - event for ebs.csi.aws.comhm6kv: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-8a9b18e2-2fdb-42f8-8097-4f86f8076203 May 11 01:10:15.753: INFO: At 2022-05-11 01:04:51 +0000 UTC - event for pod-47dd5244-65cc-4afc-8cb7-141bcd1257af: {default-scheduler } Scheduled: Successfully assigned volume-expand-4949/pod-47dd5244-65cc-4afc-8cb7-141bcd1257af to ip-172-20-58-208.us-west-2.compute.internal May 11 01:10:15.753: INFO: At 2022-05-11 01:04:53 +0000 UTC - event for pod-47dd5244-65cc-4afc-8cb7-141bcd1257af: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-8a9b18e2-2fdb-42f8-8097-4f86f8076203" May 11 01:10:15.753: INFO: At 2022-05-11 01:04:55 +0000 UTC - event for pod-47dd5244-65cc-4afc-8cb7-141bcd1257af: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-8a9b18e2-2fdb-42f8-8097-4f86f8076203" : rpc error: code = Internal desc = could not format "/dev/nvme5n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a9b18e2-2fdb-42f8-8097-4f86f8076203/globalmount": failed to get disk format of disk /dev/nvme5n1: signal: segmentation fault (core dumped) May 11 01:10:15.753: INFO: At 2022-05-11 01:06:54 +0000 UTC - event for pod-47dd5244-65cc-4afc-8cb7-141bcd1257af: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-jb4mz]: timed out waiting for the condition May 11 01:10:15.813: INFO: POD NODE PHASE GRACE CONDITIONS May 11 01:10:15.813: INFO: May 11 01:10:15.879: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 01:10:15.940: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 14492 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 105 lines ... [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould resize volume when PVC is edited while pod is using it [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246[0m [91mWhile creating pods for resizing Unexpected error: <*errors.errorString | 0xc000a4ac10>: { s: "pod \"pod-47dd5244-65cc-4afc-8cb7-141bcd1257af\" is not Running: timed out waiting for the condition", } pod "pod-47dd5244-65cc-4afc-8cb7-141bcd1257af" is not Running: timed out waiting for the condition occurred[0m ... skipping 142 lines ... May 11 01:11:12.657: INFO: At 2022-05-11 01:01:08 +0000 UTC - event for ebs.csi.aws.comhfh5f: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 01:11:12.657: INFO: At 2022-05-11 01:01:08 +0000 UTC - event for ebs.csi.aws.comhfh5f: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-438/ebs.csi.aws.comhfh5f" May 11 01:11:12.657: INFO: At 2022-05-11 01:01:08 +0000 UTC - event for ebs.csi.aws.comhfh5f: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 01:11:12.657: INFO: At 2022-05-11 01:01:11 +0000 UTC - event for ebs.csi.aws.comhfh5f: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-806dd228-2c66-4ddf-ab12-d14bd6664724 May 11 01:11:12.657: INFO: At 2022-05-11 01:01:12 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-438/pod-55a64f41-8696-48c2-90b2-8c8067b04c60 to ip-172-20-58-208.us-west-2.compute.internal May 11 01:11:12.657: INFO: At 2022-05-11 01:01:14 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-806dd228-2c66-4ddf-ab12-d14bd6664724" May 11 01:11:12.657: INFO: At 2022-05-11 01:01:28 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-806dd228-2c66-4ddf-ab12-d14bd6664724" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-806dd228-2c66-4ddf-ab12-d14bd6664724/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped) May 11 01:11:12.657: INFO: At 2022-05-11 01:03:15 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-jw2jl]: timed out waiting for the condition May 11 01:11:12.657: INFO: At 2022-05-11 01:05:33 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-jw2jl volume1]: timed out waiting for the condition May 11 01:11:12.657: INFO: At 2022-05-11 01:08:04 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-jw2jl]: error processing PVC fsgroupchangepolicy-438/ebs.csi.aws.comhfh5f: PVC is being deleted May 11 01:11:12.657: INFO: At 2022-05-11 01:08:18 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-jw2jl volume1]: error processing PVC fsgroupchangepolicy-438/ebs.csi.aws.comhfh5f: PVC is being deleted May 11 01:11:12.717: INFO: POD NODE PHASE GRACE CONDITIONS May 11 01:11:12.717: INFO: pod-55a64f41-8696-48c2-90b2-8c8067b04c60 ip-172-20-58-208.us-west-2.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:01:12 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:01:12 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:01:12 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:01:12 +0000 UTC }] May 11 01:11:12.717: INFO: May 11 01:11:12.780: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 01:11:12.840: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 14492 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 106 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1m(OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208[0m [91mUnexpected error: <*errors.errorString | 0xc0001f9520>: { s: "pod \"pod-55a64f41-8696-48c2-90b2-8c8067b04c60\" is not Running: timed out waiting for the condition", } pod "pod-55a64f41-8696-48c2-90b2-8c8067b04c60" is not Running: timed out waiting for the condition occurred[0m ... skipping 92 lines ... May 11 01:13:47.640: INFO: At 2022-05-11 01:03:43 +0000 UTC - event for ebs.csi.aws.comh9v8t: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 01:13:47.640: INFO: At 2022-05-11 01:03:43 +0000 UTC - event for ebs.csi.aws.comh9v8t: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-5568/ebs.csi.aws.comh9v8t" May 11 01:13:47.640: INFO: At 2022-05-11 01:03:43 +0000 UTC - event for ebs.csi.aws.comh9v8t: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 01:13:47.640: INFO: At 2022-05-11 01:03:46 +0000 UTC - event for ebs.csi.aws.comh9v8t: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-3ba11ba1-e9d0-4c83-8574-d5f5d3b3fbb8 May 11 01:13:47.640: INFO: At 2022-05-11 01:03:47 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-5568/pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34 to ip-172-20-58-208.us-west-2.compute.internal May 11 01:13:47.640: INFO: At 2022-05-11 01:03:49 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-3ba11ba1-e9d0-4c83-8574-d5f5d3b3fbb8" May 11 01:13:47.640: INFO: At 2022-05-11 01:03:55 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-3ba11ba1-e9d0-4c83-8574-d5f5d3b3fbb8" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3ba11ba1-e9d0-4c83-8574-d5f5d3b3fbb8/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped) May 11 01:13:47.640: INFO: At 2022-05-11 01:05:50 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-rsmtb]: timed out waiting for the condition May 11 01:13:47.640: INFO: At 2022-05-11 01:10:37 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-rsmtb]: error processing PVC fsgroupchangepolicy-5568/ebs.csi.aws.comh9v8t: PVC is being deleted May 11 01:13:47.640: INFO: At 2022-05-11 01:11:40 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-rsmtb volume1]: error processing PVC fsgroupchangepolicy-5568/ebs.csi.aws.comh9v8t: PVC is being deleted May 11 01:13:47.705: INFO: POD NODE PHASE GRACE CONDITIONS May 11 01:13:47.705: INFO: pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34 ip-172-20-58-208.us-west-2.compute.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:03:47 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:03:47 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:03:47 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:03:47 +0000 UTC }] May 11 01:13:47.705: INFO: May 11 01:13:47.770: INFO: Logging node info for node ip-172-20-109-66.us-west-2.compute.internal May 11 01:13:47.835: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal 47975c71-52c5-4a1c-b17e-c2a836d06284 14492 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>} BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>} BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},} ... skipping 106 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1m(Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208[0m [91mUnexpected error: <*errors.errorString | 0xc000ac0a10>: { s: "pod \"pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34\" is not Running: timed out waiting for the condition", } pod "pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34" is not Running: timed out waiting for the condition occurred[0m ... skipping 59 lines ... May 11 01:15:06.296: INFO: Waiting for pod external-injector to disappear May 11 01:15:06.359: INFO: Pod external-injector still exists May 11 01:15:08.297: INFO: Waiting for pod external-injector to disappear May 11 01:15:08.359: INFO: Pod external-injector still exists May 11 01:15:10.296: INFO: Waiting for pod external-injector to disappear May 11 01:15:10.357: INFO: Pod external-injector no longer exists May 11 01:15:10.357: FAIL: Failed to create injector pod: timed out waiting for the condition Full Stack Trace k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3() /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 +0x485 github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes.TestEBSCSI(0x4082b9?) /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes/e2e_test.go:82 +0x311 ... skipping 17 lines ... May 11 01:15:25.925: INFO: At 2022-05-11 01:09:59 +0000 UTC - event for ebs.csi.aws.com4rfmk: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding May 11 01:15:25.925: INFO: At 2022-05-11 01:09:59 +0000 UTC - event for ebs.csi.aws.com4rfmk: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-184/ebs.csi.aws.com4rfmk" May 11 01:15:25.926: INFO: At 2022-05-11 01:10:00 +0000 UTC - event for ebs.csi.aws.com4rfmk: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator May 11 01:15:25.926: INFO: At 2022-05-11 01:10:03 +0000 UTC - event for ebs.csi.aws.com4rfmk: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2 May 11 01:15:25.926: INFO: At 2022-05-11 01:10:04 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned volume-184/external-injector to ip-172-20-58-208.us-west-2.compute.internal May 11 01:15:25.926: INFO: At 2022-05-11 01:10:07 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2" May 11 01:15:25.926: INFO: At 2022-05-11 01:10:19 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2" : rpc error: code = Internal desc = Could not mount "/dev/nvme1n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2/6bca25ca-be09-45e2-8996-e9f65094fa70": mount failed: exec: "mount": executable file not found in $PATH Mounting command: mount Mounting arguments: -o bind /dev/nvme1n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2/6bca25ca-be09-45e2-8996-e9f65094fa70 Output: May 11 01:15:25.926: INFO: At 2022-05-11 01:12:06 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[default-token-f77jn external-volume-0]: timed out waiting for the condition May 11 01:15:25.987: INFO: POD NODE PHASE GRACE CONDITIONS May 11 01:15:25.987: INFO: ... skipping 108 lines ... [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (block volmode)] volumes [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [91m[1mshould store data [It][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159[0m [91mMay 11 01:15:10.357: Failed to create injector pod: timed out waiting for the condition[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 [90m------------------------------[0m [36mS[0m [90m------------------------------[0m [BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath ... skipping 47 lines ... [36m[1mS [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds][0m External Storage [Driver: ebs.csi.aws.com] [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175[0m [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50[0m [36m[1mshould fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach][0m [90m/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240[0m [36mDriver ebs.csi.aws.com doesn't support ntfs -- skipping[0m /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:121 [90m------------------------------[0m [36mS[0m[36mS[0m{"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:169","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Entrypoint received interrupt: terminated","severity":"error","time":"2022-05-11T01:16:05Z"} ++ early_exit_handler ++ '[' -n 171 ']' ++ kill -TERM 171 ++ cleanup_dind ++ [[ true == \t\r\u\e ]] ++ echo 'Cleaning up after docker' ... skipping 5 lines ...