This job view page is being replaced by Spyglass soon. Check out the new job view.
PRtorredil: Use minimal base image for linux builds
ResultABORTED
Tests 0 failed / 0 succeeded
Started2022-05-11 00:02
Elapsed1h28m
Revisiona1ab90890e0e26c98677e8b529abdd7d8fb20f0d
Refs 1233

No Test Failures!


Error lines from build-log.txt

... skipping 588 lines ...
## Validating cluster test-cluster-27998.k8s.local
#
Using cluster from kubectl context: test-cluster-27998.k8s.local

Validating cluster test-cluster-27998.k8s.local

W0511 00:04:59.174114    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:05:09.205711    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:05:19.227407    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:05:29.262282    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:05:39.293542    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:05:49.327311    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:05:59.377377    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:06:09.414259    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:06:19.460653    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:06:30.657616    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: Get "https://api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com/api/v1/nodes": dial tcp: lookup api-test-cluster-27998-k8-hqi5rv-626490162.us-west-2.elb.amazonaws.com on 10.63.240.10:53: no such host
W0511 00:06:53.361729    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: an error on the server ("") has prevented the request from succeeding (get nodes)
W0511 00:07:14.943523    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: an error on the server ("") has prevented the request from succeeding (get nodes)
W0511 00:07:36.525369    9196 validate_cluster.go:173] (will retry): unexpected error during validation: error listing nodes: an error on the server ("") has prevented the request from succeeding (get nodes)
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-2a	Master	t3.medium	1	1	us-west-2a
nodes-us-west-2a	Node	c5.large	1	1	us-west-2a
nodes-us-west-2b	Node	c5.large	1	1	us-west-2b
nodes-us-west-2c	Node	c5.large	1	1	us-west-2c
... skipping 9 lines ...
Machine	i-06679b4f8ef30e496				machine "i-06679b4f8ef30e496" has not yet joined cluster
Node	ip-172-20-52-249.us-west-2.compute.internal	node "ip-172-20-52-249.us-west-2.compute.internal" of role "master" is not ready
Pod	kube-system/coredns-8f5559c9b-qgzpn		system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending
Pod	kube-system/coredns-autoscaler-6f594f4c58-gb99g	system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending
Pod	kube-system/dns-controller-5d59c585d8-nfvwz	system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending

Validation Failed
W0511 00:07:50.496247    9196 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-2a	Master	t3.medium	1	1	us-west-2a
nodes-us-west-2a	Node	c5.large	1	1	us-west-2a
nodes-us-west-2b	Node	c5.large	1	1	us-west-2b
... skipping 9 lines ...
Machine	i-057238a8395634bb6				machine "i-057238a8395634bb6" has not yet joined cluster
Machine	i-06679b4f8ef30e496				machine "i-06679b4f8ef30e496" has not yet joined cluster
Pod	kube-system/coredns-8f5559c9b-qgzpn		system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending
Pod	kube-system/coredns-autoscaler-6f594f4c58-gb99g	system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending
Pod	kube-system/dns-controller-5d59c585d8-nfvwz	system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending

Validation Failed
W0511 00:08:02.674657    9196 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-2a	Master	t3.medium	1	1	us-west-2a
nodes-us-west-2a	Node	c5.large	1	1	us-west-2a
nodes-us-west-2b	Node	c5.large	1	1	us-west-2b
... skipping 10 lines ...
Machine	i-06679b4f8ef30e496								machine "i-06679b4f8ef30e496" has not yet joined cluster
Pod	kube-system/coredns-8f5559c9b-qgzpn						system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending
Pod	kube-system/coredns-autoscaler-6f594f4c58-gb99g					system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending
Pod	kube-system/dns-controller-5d59c585d8-nfvwz					system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending
Pod	kube-system/etcd-manager-main-ip-172-20-52-249.us-west-2.compute.internal	system-cluster-critical pod "etcd-manager-main-ip-172-20-52-249.us-west-2.compute.internal" is pending

Validation Failed
W0511 00:08:14.942216    9196 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-2a	Master	t3.medium	1	1	us-west-2a
nodes-us-west-2a	Node	c5.large	1	1	us-west-2a
nodes-us-west-2b	Node	c5.large	1	1	us-west-2b
... skipping 9 lines ...
Machine	i-057238a8395634bb6				machine "i-057238a8395634bb6" has not yet joined cluster
Machine	i-06679b4f8ef30e496				machine "i-06679b4f8ef30e496" has not yet joined cluster
Pod	kube-system/coredns-8f5559c9b-qgzpn		system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending
Pod	kube-system/coredns-autoscaler-6f594f4c58-gb99g	system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending
Pod	kube-system/dns-controller-5d59c585d8-nfvwz	system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending

Validation Failed
W0511 00:08:27.069508    9196 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-2a	Master	t3.medium	1	1	us-west-2a
nodes-us-west-2a	Node	c5.large	1	1	us-west-2a
nodes-us-west-2b	Node	c5.large	1	1	us-west-2b
... skipping 11 lines ...
Node	ip-172-20-52-249.us-west-2.compute.internal	master "ip-172-20-52-249.us-west-2.compute.internal" is missing kube-apiserver pod
Pod	kube-system/coredns-8f5559c9b-qgzpn		system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending
Pod	kube-system/coredns-autoscaler-6f594f4c58-gb99g	system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending
Pod	kube-system/dns-controller-5d59c585d8-nfvwz	system-cluster-critical pod "dns-controller-5d59c585d8-nfvwz" is pending
Pod	kube-system/kops-controller-k2pcb		system-node-critical pod "kops-controller-k2pcb" is pending

Validation Failed
W0511 00:08:39.261442    9196 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-2a	Master	t3.medium	1	1	us-west-2a
nodes-us-west-2a	Node	c5.large	1	1	us-west-2a
nodes-us-west-2b	Node	c5.large	1	1	us-west-2b
... skipping 8 lines ...
Machine	i-042bf187445039118				machine "i-042bf187445039118" has not yet joined cluster
Machine	i-057238a8395634bb6				machine "i-057238a8395634bb6" has not yet joined cluster
Machine	i-06679b4f8ef30e496				machine "i-06679b4f8ef30e496" has not yet joined cluster
Pod	kube-system/coredns-8f5559c9b-qgzpn		system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending
Pod	kube-system/coredns-autoscaler-6f594f4c58-gb99g	system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending

Validation Failed
W0511 00:08:51.438174    9196 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-2a	Master	t3.medium	1	1	us-west-2a
nodes-us-west-2a	Node	c5.large	1	1	us-west-2a
nodes-us-west-2b	Node	c5.large	1	1	us-west-2b
... skipping 12 lines ...
Node	ip-172-20-58-208.us-west-2.compute.internal				node "ip-172-20-58-208.us-west-2.compute.internal" of role "node" is not ready
Node	ip-172-20-84-177.us-west-2.compute.internal				node "ip-172-20-84-177.us-west-2.compute.internal" of role "node" is not ready
Pod	kube-system/coredns-8f5559c9b-qgzpn					system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is pending
Pod	kube-system/coredns-autoscaler-6f594f4c58-gb99g				system-cluster-critical pod "coredns-autoscaler-6f594f4c58-gb99g" is pending
Pod	kube-system/kube-proxy-ip-172-20-84-177.us-west-2.compute.internal	system-node-critical pod "kube-proxy-ip-172-20-84-177.us-west-2.compute.internal" is pending

Validation Failed
W0511 00:09:03.991432    9196 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-2a	Master	t3.medium	1	1	us-west-2a
nodes-us-west-2a	Node	c5.large	1	1	us-west-2a
nodes-us-west-2b	Node	c5.large	1	1	us-west-2b
... skipping 8 lines ...

VALIDATION ERRORS
KIND	NAME					MESSAGE
Pod	kube-system/coredns-8f5559c9b-htw9q	system-cluster-critical pod "coredns-8f5559c9b-htw9q" is not ready (coredns)
Pod	kube-system/coredns-8f5559c9b-qgzpn	system-cluster-critical pod "coredns-8f5559c9b-qgzpn" is not ready (coredns)

Validation Failed
W0511 00:09:16.417888    9196 validate_cluster.go:221] (will retry): cluster not yet healthy
INSTANCE GROUPS
NAME			ROLE	MACHINETYPE	MIN	MAX	SUBNETS
master-us-west-2a	Master	t3.medium	1	1	us-west-2a
nodes-us-west-2a	Node	c5.large	1	1	us-west-2a
nodes-us-west-2b	Node	c5.large	1	1	us-west-2b
... skipping 196 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:278

    Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
... skipping 101 lines ...
May 11 00:11:07.093: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-514-e2e-scxhzv5
STEP: creating a claim
May 11 00:11:07.161: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod pod-subpath-test-dynamicpv-rqfj
STEP: Creating a pod to test atomic-volume-subpath
May 11 00:11:07.373: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-rqfj" in namespace "provisioning-514" to be "Succeeded or Failed"
May 11 00:11:07.435: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 62.196096ms
May 11 00:11:09.499: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 2.125518858s
May 11 00:11:11.562: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4.188253688s
May 11 00:11:13.625: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 6.252125131s
May 11 00:11:15.689: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 8.31547365s
May 11 00:11:17.753: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 10.379992196s
... skipping 134 lines ...
May 11 00:15:56.365: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.992107235s
May 11 00:15:58.428: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.055160938s
May 11 00:16:00.492: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.118662638s
May 11 00:16:02.557: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.184176036s
May 11 00:16:04.620: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.247179641s
May 11 00:16:06.683: INFO: Pod "pod-subpath-test-dynamicpv-rqfj": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.310167183s
May 11 00:16:08.830: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-rqfj" container "init-volume-dynamicpv-rqfj": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-rqfj)
May 11 00:16:08.894: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-rqfj" container "test-container-subpath-dynamicpv-rqfj": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-rqfj)
STEP: delete the pod
May 11 00:16:08.961: INFO: Waiting for pod pod-subpath-test-dynamicpv-rqfj to disappear
May 11 00:16:09.024: INFO: Pod pod-subpath-test-dynamicpv-rqfj still exists
May 11 00:16:11.024: INFO: Waiting for pod pod-subpath-test-dynamicpv-rqfj to disappear
May 11 00:16:11.087: INFO: Pod pod-subpath-test-dynamicpv-rqfj still exists
May 11 00:16:13.024: INFO: Waiting for pod pod-subpath-test-dynamicpv-rqfj to disappear
... skipping 25 lines ...
May 11 00:16:54.985: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.com6mjcs: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:16:54.985: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.com6mjcs: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-514/ebs.csi.aws.com6mjcs"
May 11 00:16:54.985: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.com6mjcs: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:16:54.985: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.com6mjcs: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-6eb1468d-e1cd-4c0c-9c70-4aeee22b6e50
May 11 00:16:54.985: INFO: At 2022-05-11 00:11:14 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {default-scheduler } Scheduled: Successfully assigned provisioning-514/pod-subpath-test-dynamicpv-rqfj to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:16:54.985: INFO: At 2022-05-11 00:11:16 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-6eb1468d-e1cd-4c0c-9c70-4aeee22b6e50" 
May 11 00:16:54.985: INFO: At 2022-05-11 00:11:23 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-6eb1468d-e1cd-4c0c-9c70-4aeee22b6e50" : rpc error: code = Internal desc = could not format "/dev/nvme2n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-6eb1468d-e1cd-4c0c-9c70-4aeee22b6e50/globalmount": failed to get disk format of disk /dev/nvme2n1: signal: segmentation fault (core dumped)
May 11 00:16:54.985: INFO: At 2022-05-11 00:13:17 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[default-token-w5mfq test-volume liveness-probe-volume]: timed out waiting for the condition
May 11 00:16:54.985: INFO: At 2022-05-11 00:15:35 +0000 UTC - event for pod-subpath-test-dynamicpv-rqfj: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-w5mfq]: timed out waiting for the condition
May 11 00:16:55.047: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:16:55.047: INFO: 
May 11 00:16:55.171: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
... skipping 101 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support file as subpath [LinuxOnly] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:230

    Unexpected error:
        <*errors.errorString | 0xc000e7b1a0>: {
            s: "expected pod \"pod-subpath-test-dynamicpv-rqfj\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-rqfj\" to be \"Succeeded or Failed\"",
        }
        expected pod "pod-subpath-test-dynamicpv-rqfj" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-rqfj" to be "Succeeded or Failed"
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (block volmode)] multiVolume [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
... skipping 57 lines ...
May 11 00:16:55.587: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comhjlvp: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:16:55.587: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comhjlvp: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "multivolume-304/ebs.csi.aws.comhjlvp"
May 11 00:16:55.587: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comhjlvp: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:16:55.587: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.comhjlvp: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-30190e47-d1f3-483a-ad66-ba7af8e91c5a
May 11 00:16:55.587: INFO: At 2022-05-11 00:11:14 +0000 UTC - event for pod-85fbad13-d472-44e2-ae52-6d9762809056: {default-scheduler } Scheduled: Successfully assigned multivolume-304/pod-85fbad13-d472-44e2-ae52-6d9762809056 to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:16:55.587: INFO: At 2022-05-11 00:11:16 +0000 UTC - event for pod-85fbad13-d472-44e2-ae52-6d9762809056: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-30190e47-d1f3-483a-ad66-ba7af8e91c5a" 
May 11 00:16:55.587: INFO: At 2022-05-11 00:11:23 +0000 UTC - event for pod-85fbad13-d472-44e2-ae52-6d9762809056: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-30190e47-d1f3-483a-ad66-ba7af8e91c5a" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-30190e47-d1f3-483a-ad66-ba7af8e91c5a/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped)
May 11 00:16:55.587: INFO: At 2022-05-11 00:13:17 +0000 UTC - event for pod-85fbad13-d472-44e2-ae52-6d9762809056: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-hht6n]: timed out waiting for the condition
May 11 00:16:55.650: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:16:55.650: INFO: 
May 11 00:16:55.773: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:16:55.836: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 2032 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:14:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:14:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:14:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:14:56 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 100 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:337

    Unexpected error:
        <*errors.errorString | 0xc000ebcd00>: {
            s: "pod \"pod-85fbad13-d472-44e2-ae52-6d9762809056\" is not Running: timed out waiting for the condition",
        }
        pod "pod-85fbad13-d472-44e2-ae52-6d9762809056" is not Running: timed out waiting for the condition
    occurred

... skipping 92 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Inline-volume (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240

    Driver "ebs.csi.aws.com" does not support volume type "InlineVolume" - skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
... skipping 134 lines ...
May 11 00:16:59.973: INFO: At 2022-05-11 00:11:08 +0000 UTC - event for ebs.csi.aws.coms4bcd: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:16:59.973: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.com4vnzb: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970
May 11 00:16:59.973: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.coms4bcd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-4c803fa3-fdb4-4375-9b02-f901bc788715
May 11 00:16:59.973: INFO: At 2022-05-11 00:11:14 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {default-scheduler } Scheduled: Successfully assigned multivolume-8585/pod-7fa7162b-0a8d-488f-a29e-29e409853fe8 to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:16:59.973: INFO: At 2022-05-11 00:11:16 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-4c803fa3-fdb4-4375-9b02-f901bc788715" 
May 11 00:16:59.973: INFO: At 2022-05-11 00:11:18 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970" 
May 11 00:16:59.973: INFO: At 2022-05-11 00:11:19 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-4c803fa3-fdb4-4375-9b02-f901bc788715" : rpc error: code = Internal desc = Could not mount "/dev/nvme3n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-4c803fa3-fdb4-4375-9b02-f901bc788715/423f46a6-d11e-4748-9d4f-bad967b58d15": mount failed: exec: "mount": executable file not found in $PATH
Mounting command: mount
Mounting arguments:  -o bind /dev/nvme3n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-4c803fa3-fdb4-4375-9b02-f901bc788715/423f46a6-d11e-4748-9d4f-bad967b58d15
Output: 
May 11 00:16:59.973: INFO: At 2022-05-11 00:11:20 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970" : rpc error: code = Internal desc = Could not mount "/dev/nvme4n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970/423f46a6-d11e-4748-9d4f-bad967b58d15": mount failed: exec: "mount": executable file not found in $PATH
Mounting command: mount
Mounting arguments:  -o bind /dev/nvme4n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-121dd4fc-7460-45a7-8b7a-f2f2bd897970/423f46a6-d11e-4748-9d4f-bad967b58d15
Output: 
May 11 00:16:59.973: INFO: At 2022-05-11 00:13:17 +0000 UTC - event for pod-7fa7162b-0a8d-488f-a29e-29e409853fe8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[default-token-gzl5g volume1 volume2]: timed out waiting for the condition
May 11 00:17:00.033: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:17:00.033: INFO: 
... skipping 103 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node [LinuxOnly] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:134

    Unexpected error:
        <*errors.errorString | 0xc00095ce20>: {
            s: "pod \"pod-7fa7162b-0a8d-488f-a29e-29e409853fe8\" is not Running: timed out waiting for the condition",
        }
        pod "pod-7fa7162b-0a8d-488f-a29e-29e409853fe8" is not Running: timed out waiting for the condition
    occurred

... skipping 83 lines ...
May 11 00:22:45.570: INFO: At 2022-05-11 00:16:59 +0000 UTC - event for ebs.csi.aws.comn2r9v: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:22:45.570: INFO: At 2022-05-11 00:16:59 +0000 UTC - event for ebs.csi.aws.comn2r9v: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "multivolume-6237/ebs.csi.aws.comn2r9v"
May 11 00:22:45.570: INFO: At 2022-05-11 00:16:59 +0000 UTC - event for ebs.csi.aws.comn2r9v: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:22:45.570: INFO: At 2022-05-11 00:17:02 +0000 UTC - event for ebs.csi.aws.comn2r9v: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-72673e89-5407-41fa-a39b-a191c254a6ba
May 11 00:22:45.570: INFO: At 2022-05-11 00:17:03 +0000 UTC - event for pod-34cc14a3-72d5-4310-9333-6e3a91b006ba: {default-scheduler } Scheduled: Successfully assigned multivolume-6237/pod-34cc14a3-72d5-4310-9333-6e3a91b006ba to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:22:45.570: INFO: At 2022-05-11 00:17:05 +0000 UTC - event for pod-34cc14a3-72d5-4310-9333-6e3a91b006ba: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-72673e89-5407-41fa-a39b-a191c254a6ba" 
May 11 00:22:45.570: INFO: At 2022-05-11 00:17:11 +0000 UTC - event for pod-34cc14a3-72d5-4310-9333-6e3a91b006ba: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-72673e89-5407-41fa-a39b-a191c254a6ba" : rpc error: code = Internal desc = Could not mount "/dev/nvme2n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-72673e89-5407-41fa-a39b-a191c254a6ba/247e98b0-7d5d-4103-949d-df4037c39504": mount failed: exec: "mount": executable file not found in $PATH
Mounting command: mount
Mounting arguments:  -o bind /dev/nvme2n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-72673e89-5407-41fa-a39b-a191c254a6ba/247e98b0-7d5d-4103-949d-df4037c39504
Output: 
May 11 00:22:45.570: INFO: At 2022-05-11 00:19:06 +0000 UTC - event for pod-34cc14a3-72d5-4310-9333-6e3a91b006ba: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-d5259 volume1]: timed out waiting for the condition
May 11 00:22:45.629: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:22:45.629: INFO: 
... skipping 103 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single volume from pods on the same node [LinuxOnly] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:312

    Unexpected error:
        <*errors.errorString | 0xc00106eb20>: {
            s: "pod \"pod-34cc14a3-72d5-4310-9333-6e3a91b006ba\" is not Running: timed out waiting for the condition",
        }
        pod "pod-34cc14a3-72d5-4310-9333-6e3a91b006ba" is not Running: timed out waiting for the condition
    occurred

... skipping 38 lines ...
May 11 00:16:57.643: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-3005-e2e-sct78gp
STEP: creating a claim
May 11 00:16:57.706: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod pod-subpath-test-dynamicpv-zs9l
STEP: Creating a pod to test subpath
May 11 00:16:57.895: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-zs9l" in namespace "provisioning-3005" to be "Succeeded or Failed"
May 11 00:16:57.958: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 62.178339ms
May 11 00:17:00.021: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 2.12544992s
May 11 00:17:02.084: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4.188246897s
May 11 00:17:04.147: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 6.251271279s
May 11 00:17:06.210: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 8.314578544s
May 11 00:17:08.273: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 10.377556447s
... skipping 134 lines ...
May 11 00:21:46.938: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.042591454s
May 11 00:21:49.001: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.105554944s
May 11 00:21:51.065: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.169301106s
May 11 00:21:53.141: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.24577554s
May 11 00:21:55.205: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.309075882s
May 11 00:21:57.268: INFO: Pod "pod-subpath-test-dynamicpv-zs9l": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.372099604s
May 11 00:21:59.412: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-zs9l" container "init-volume-dynamicpv-zs9l": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-zs9l)
May 11 00:21:59.476: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-zs9l" container "test-init-subpath-dynamicpv-zs9l": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-zs9l)
May 11 00:21:59.540: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-zs9l" container "test-container-subpath-dynamicpv-zs9l": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-zs9l)
STEP: delete the pod
May 11 00:21:59.604: INFO: Waiting for pod pod-subpath-test-dynamicpv-zs9l to disappear
May 11 00:21:59.667: INFO: Pod pod-subpath-test-dynamicpv-zs9l still exists
May 11 00:22:01.667: INFO: Waiting for pod pod-subpath-test-dynamicpv-zs9l to disappear
May 11 00:22:01.735: INFO: Pod pod-subpath-test-dynamicpv-zs9l still exists
May 11 00:22:03.669: INFO: Waiting for pod pod-subpath-test-dynamicpv-zs9l to disappear
... skipping 25 lines ...
May 11 00:22:45.619: INFO: At 2022-05-11 00:16:57 +0000 UTC - event for ebs.csi.aws.coms8h4n: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:22:45.619: INFO: At 2022-05-11 00:16:57 +0000 UTC - event for ebs.csi.aws.coms8h4n: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-3005/ebs.csi.aws.coms8h4n"
May 11 00:22:45.619: INFO: At 2022-05-11 00:16:57 +0000 UTC - event for ebs.csi.aws.coms8h4n: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:22:45.619: INFO: At 2022-05-11 00:17:01 +0000 UTC - event for ebs.csi.aws.coms8h4n: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-04c19764-71c4-4b24-be43-4a1315e17e34
May 11 00:22:45.619: INFO: At 2022-05-11 00:17:01 +0000 UTC - event for pod-subpath-test-dynamicpv-zs9l: {default-scheduler } Scheduled: Successfully assigned provisioning-3005/pod-subpath-test-dynamicpv-zs9l to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:22:45.619: INFO: At 2022-05-11 00:17:03 +0000 UTC - event for pod-subpath-test-dynamicpv-zs9l: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-04c19764-71c4-4b24-be43-4a1315e17e34" 
May 11 00:22:45.619: INFO: At 2022-05-11 00:17:09 +0000 UTC - event for pod-subpath-test-dynamicpv-zs9l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-04c19764-71c4-4b24-be43-4a1315e17e34" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-04c19764-71c4-4b24-be43-4a1315e17e34/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped)
May 11 00:22:45.619: INFO: At 2022-05-11 00:19:04 +0000 UTC - event for pod-subpath-test-dynamicpv-zs9l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-89v5h]: timed out waiting for the condition
May 11 00:22:45.689: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:22:45.689: INFO: 
May 11 00:22:45.753: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:22:45.816: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 3192 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:19:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:19:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:19:56 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:19:56 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 100 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support existing directories when readOnly specified in the volumeSource [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:399

    Unexpected error:
        <*errors.errorString | 0xc000e7a6f0>: {
            s: "expected pod \"pod-subpath-test-dynamicpv-zs9l\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-zs9l\" to be \"Succeeded or Failed\"",
        }
        expected pod "pod-subpath-test-dynamicpv-zs9l" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-zs9l" to be "Succeeded or Failed"
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742
------------------------------
S
------------------------------
... skipping 35 lines ...

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:121
------------------------------
SS
------------------------------
External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (delayed binding)] topology 
  should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192

[BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (delayed binding)] topology
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185
STEP: Creating a kubernetes client
May 11 00:22:47.905: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192
May 11 00:22:48.265: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2c]
May 11 00:22:48.265: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2a]
May 11 00:22:48.266: INFO: Creating storage class object and pvc object for driver - sc: &StorageClass{ObjectMeta:{topology-5303-e2e-scs5479      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{},ReclaimPolicy:nil,MountOptions:[],AllowVolumeExpansion:nil,VolumeBindingMode:*WaitForFirstConsumer,AllowedTopologies:[]TopologySelectorTerm{{[{topology.ebs.csi.aws.com/zone [us-west-2a]}]},},}, pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- topology-5303    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*topology-5303-e2e-scs5479,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},}
STEP: Creating sc
STEP: Creating pvc
... skipping 69 lines ...
May 11 00:22:50.453: INFO: At 2022-05-11 00:17:02 +0000 UTC - event for ebs.csi.aws.commlpwc: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:22:50.453: INFO: At 2022-05-11 00:17:06 +0000 UTC - event for ebs.csi.aws.com5zcnv: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-83c349f4-0ebe-49c7-ab2b-f77cedfb8371
May 11 00:22:50.453: INFO: At 2022-05-11 00:17:06 +0000 UTC - event for ebs.csi.aws.commlpwc: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-10002f2b-9ab8-43e3-add1-235c361b22ba
May 11 00:22:50.453: INFO: At 2022-05-11 00:17:06 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {default-scheduler } Scheduled: Successfully assigned multivolume-5490/pod-81676b9b-64a2-4e2a-be69-fe929301d43d to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:22:50.453: INFO: At 2022-05-11 00:17:08 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-10002f2b-9ab8-43e3-add1-235c361b22ba" 
May 11 00:22:50.453: INFO: At 2022-05-11 00:17:08 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-83c349f4-0ebe-49c7-ab2b-f77cedfb8371" 
May 11 00:22:50.453: INFO: At 2022-05-11 00:17:10 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-10002f2b-9ab8-43e3-add1-235c361b22ba" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-10002f2b-9ab8-43e3-add1-235c361b22ba/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped)
May 11 00:22:50.453: INFO: At 2022-05-11 00:17:10 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-83c349f4-0ebe-49c7-ab2b-f77cedfb8371" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-83c349f4-0ebe-49c7-ab2b-f77cedfb8371/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped)
May 11 00:22:50.453: INFO: At 2022-05-11 00:19:09 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume2 volume1], unattached volumes=[volume2 default-token-fvrwj volume1]: timed out waiting for the condition
May 11 00:22:50.453: INFO: At 2022-05-11 00:21:25 +0000 UTC - event for pod-81676b9b-64a2-4e2a-be69-fe929301d43d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1 volume2], unattached volumes=[volume1 volume2 default-token-fvrwj]: timed out waiting for the condition
May 11 00:22:50.513: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:22:50.513: INFO: 
May 11 00:22:50.573: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
... skipping 101 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (filesystem volmode)] multiVolume [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should access to two volumes with the same volume mode and retain data across pod recreation on the same node [LinuxOnly] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:134

    Unexpected error:
        <*errors.errorString | 0xc000f10a70>: {
            s: "pod \"pod-81676b9b-64a2-4e2a-be69-fe929301d43d\" is not Running: timed out waiting for the condition",
        }
        pod "pod-81676b9b-64a2-4e2a-be69-fe929301d43d" is not Running: timed out waiting for the condition
    occurred

... skipping 16 lines ...
May 11 00:11:07.617: INFO: Creating resource for dynamic PV
May 11 00:11:07.617: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass snapshotting-8875-e2e-scr87p2
STEP: creating a claim
May 11 00:11:07.682: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: [init] starting a pod to use the claim
May 11 00:11:07.877: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-4hxlq" in namespace "snapshotting-8875" to be "Succeeded or Failed"
May 11 00:11:07.940: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 63.589663ms
May 11 00:11:10.004: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 2.127461557s
May 11 00:11:12.068: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 4.191260691s
May 11 00:11:14.132: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 6.255068183s
May 11 00:11:16.197: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 8.320105573s
May 11 00:11:18.262: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 10.384943045s
... skipping 424 lines ...
May 11 00:25:55.989: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m48.112249496s
May 11 00:25:58.056: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m50.17925948s
May 11 00:26:00.124: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m52.247344554s
May 11 00:26:02.192: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m54.315108137s
May 11 00:26:04.259: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m56.382144645s
May 11 00:26:06.327: INFO: Pod "pvc-snapshottable-tester-4hxlq": Phase="Pending", Reason="", readiness=false. Elapsed: 14m58.449662569s
May 11 00:26:08.402: INFO: Error getting logs for pod pvc-snapshottable-tester-4hxlq: the server rejected our request for an unknown reason (get pods pvc-snapshottable-tester-4hxlq)
May 11 00:26:08.402: INFO: Deleting pod "pvc-snapshottable-tester-4hxlq" in namespace "snapshotting-8875"
May 11 00:26:08.471: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-4hxlq" to be fully deleted
[AfterEach] volume snapshot controller
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:223
STEP: Deleting pvc
May 11 00:26:18.771: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comb4jmx"
... skipping 14 lines ...
May 11 00:26:54.532: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comb4jmx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:26:54.532: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comb4jmx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "snapshotting-8875/ebs.csi.aws.comb4jmx"
May 11 00:26:54.532: INFO: At 2022-05-11 00:11:07 +0000 UTC - event for ebs.csi.aws.comb4jmx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:26:54.532: INFO: At 2022-05-11 00:11:13 +0000 UTC - event for ebs.csi.aws.comb4jmx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-568e9664-9e2e-4c06-b5d0-a90058448d83
May 11 00:26:54.532: INFO: At 2022-05-11 00:11:14 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {default-scheduler } Scheduled: Successfully assigned snapshotting-8875/pvc-snapshottable-tester-4hxlq to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:26:54.532: INFO: At 2022-05-11 00:11:18 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-568e9664-9e2e-4c06-b5d0-a90058448d83" 
May 11 00:26:54.532: INFO: At 2022-05-11 00:11:18 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-568e9664-9e2e-4c06-b5d0-a90058448d83" : rpc error: code = Internal desc = could not format "/dev/nvme5n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-568e9664-9e2e-4c06-b5d0-a90058448d83/globalmount": failed to get disk format of disk /dev/nvme5n1: signal: segmentation fault (core dumped)
May 11 00:26:54.532: INFO: At 2022-05-11 00:13:17 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[my-volume default-token-6mq28]: timed out waiting for the condition
May 11 00:26:54.532: INFO: At 2022-05-11 00:15:34 +0000 UTC - event for pvc-snapshottable-tester-4hxlq: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[default-token-6mq28 my-volume]: timed out waiting for the condition
May 11 00:26:54.598: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:26:54.598: INFO: 
May 11 00:26:54.665: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
... skipping 110 lines ...
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108
       [BeforeEach]
      /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:227
        should check snapshot fields, check restore correctly works after modifying source data, check deletion
        /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:245

        Unexpected error:
            <*errors.errorString | 0xc000854720>: {
                s: "Gave up after waiting 15m0s for pod \"pvc-snapshottable-tester-4hxlq\" to be \"Succeeded or Failed\"",
            }
            Gave up after waiting 15m0s for pod "pvc-snapshottable-tester-4hxlq" to be "Succeeded or Failed"
        occurred

        /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:675
------------------------------
SS
------------------------------
... skipping 65 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240

    Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185
STEP: Creating a kubernetes client
May 11 00:22:52.503: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
[It] should fail if non-existent subpath is outside the volume [Slow][LinuxOnly]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267
May 11 00:22:52.799: INFO: Creating resource for dynamic PV
May 11 00:22:52.800: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-2924-e2e-scrj6k6
STEP: creating a claim
May 11 00:22:52.861: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod pod-subpath-test-dynamicpv-cvn4
STEP: Checking for subpath error in container status
May 11 00:27:53.237: INFO: Deleting pod "pod-subpath-test-dynamicpv-cvn4" in namespace "provisioning-2924"
May 11 00:27:53.297: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-cvn4" to be fully deleted
STEP: Deleting pod
May 11 00:27:59.418: INFO: Deleting pod "pod-subpath-test-dynamicpv-cvn4" in namespace "provisioning-2924"
STEP: Deleting pvc
May 11 00:27:59.599: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comnfncb"
... skipping 9 lines ...
May 11 00:28:09.967: INFO: At 2022-05-11 00:22:52 +0000 UTC - event for ebs.csi.aws.comnfncb: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:28:09.967: INFO: At 2022-05-11 00:22:53 +0000 UTC - event for ebs.csi.aws.comnfncb: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-2924/ebs.csi.aws.comnfncb"
May 11 00:28:09.967: INFO: At 2022-05-11 00:22:53 +0000 UTC - event for ebs.csi.aws.comnfncb: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:28:09.967: INFO: At 2022-05-11 00:22:56 +0000 UTC - event for ebs.csi.aws.comnfncb: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-b2f2a27d-535d-4769-8b8a-321c841e051b
May 11 00:28:09.967: INFO: At 2022-05-11 00:22:57 +0000 UTC - event for pod-subpath-test-dynamicpv-cvn4: {default-scheduler } Scheduled: Successfully assigned provisioning-2924/pod-subpath-test-dynamicpv-cvn4 to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:28:09.967: INFO: At 2022-05-11 00:22:58 +0000 UTC - event for pod-subpath-test-dynamicpv-cvn4: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-b2f2a27d-535d-4769-8b8a-321c841e051b" 
May 11 00:28:09.967: INFO: At 2022-05-11 00:23:01 +0000 UTC - event for pod-subpath-test-dynamicpv-cvn4: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-b2f2a27d-535d-4769-8b8a-321c841e051b" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b2f2a27d-535d-4769-8b8a-321c841e051b/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped)
May 11 00:28:09.967: INFO: At 2022-05-11 00:25:00 +0000 UTC - event for pod-subpath-test-dynamicpv-cvn4: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-xd2zw]: timed out waiting for the condition
May 11 00:28:10.027: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:28:10.027: INFO: 
May 11 00:28:10.089: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:28:10.149: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 4337 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 101 lines ...

• Failure [319.523 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267

    while waiting for subpath failure
    Unexpected error:
        <*errors.errorString | 0xc00110fa80>: {
            s: "error waiting for pod subpath error to occur: timed out waiting for the condition",
        }
        error waiting for pod subpath error to occur: timed out waiting for the condition
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:718
------------------------------
External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] volumes 
  should store data
... skipping 25 lines ...
May 11 00:27:54.811: INFO: Waiting for pod external-injector to disappear
May 11 00:27:54.874: INFO: Pod external-injector still exists
May 11 00:27:56.812: INFO: Waiting for pod external-injector to disappear
May 11 00:27:56.875: INFO: Pod external-injector still exists
May 11 00:27:58.811: INFO: Waiting for pod external-injector to disappear
May 11 00:27:58.874: INFO: Pod external-injector no longer exists
May 11 00:27:58.874: FAIL: Failed to create injector pod: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3()
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 +0x485
github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes.TestEBSCSI(0x4082b9?)
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes/e2e_test.go:82 +0x311
... skipping 17 lines ...
May 11 00:28:14.454: INFO: At 2022-05-11 00:22:48 +0000 UTC - event for ebs.csi.aws.com6p55l: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:28:14.454: INFO: At 2022-05-11 00:22:48 +0000 UTC - event for ebs.csi.aws.com6p55l: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-1461/ebs.csi.aws.com6p55l"
May 11 00:28:14.454: INFO: At 2022-05-11 00:22:48 +0000 UTC - event for ebs.csi.aws.com6p55l: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:28:14.454: INFO: At 2022-05-11 00:22:51 +0000 UTC - event for ebs.csi.aws.com6p55l: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-5485ee07-2692-42b8-a43c-499219b1905f
May 11 00:28:14.454: INFO: At 2022-05-11 00:22:52 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned volume-1461/external-injector to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:28:14.454: INFO: At 2022-05-11 00:22:56 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-5485ee07-2692-42b8-a43c-499219b1905f" 
May 11 00:28:14.454: INFO: At 2022-05-11 00:23:00 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-5485ee07-2692-42b8-a43c-499219b1905f" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5485ee07-2692-42b8-a43c-499219b1905f/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped)
May 11 00:28:14.454: INFO: At 2022-05-11 00:24:55 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 default-token-47rxx]: timed out waiting for the condition
May 11 00:28:14.516: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:28:14.516: INFO: 
May 11 00:28:14.579: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:28:14.642: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 4337 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:24:58 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 104 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] volumes
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should store data [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159

    May 11 00:27:58.874: Failed to create injector pod: timed out waiting for the condition

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186
------------------------------
SS
------------------------------
External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] subPath 
... skipping 14 lines ...
May 11 00:26:57.205: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-4055-e2e-sckg8q7
STEP: creating a claim
May 11 00:26:57.274: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod pod-subpath-test-dynamicpv-gbfd
STEP: Creating a pod to test subpath
May 11 00:26:57.479: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-gbfd" in namespace "provisioning-4055" to be "Succeeded or Failed"
May 11 00:26:57.546: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 66.366694ms
May 11 00:26:59.612: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 2.132754904s
May 11 00:27:01.681: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4.2011741s
May 11 00:27:03.749: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 6.269021154s
May 11 00:27:05.815: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 8.335909992s
May 11 00:27:07.884: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 10.404169837s
... skipping 134 lines ...
May 11 00:31:47.121: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.641371636s
May 11 00:31:49.188: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.708844019s
May 11 00:31:51.257: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.777208704s
May 11 00:31:53.325: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.845142615s
May 11 00:31:55.393: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.913922838s
May 11 00:31:57.461: INFO: Pod "pod-subpath-test-dynamicpv-gbfd": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.981766397s
May 11 00:31:59.603: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-gbfd" container "init-volume-dynamicpv-gbfd": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-gbfd)
May 11 00:31:59.671: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-gbfd" container "test-init-volume-dynamicpv-gbfd": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-gbfd)
May 11 00:31:59.740: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-gbfd" container "test-container-subpath-dynamicpv-gbfd": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-gbfd)
STEP: delete the pod
May 11 00:31:59.810: INFO: Waiting for pod pod-subpath-test-dynamicpv-gbfd to disappear
May 11 00:31:59.876: INFO: Pod pod-subpath-test-dynamicpv-gbfd still exists
May 11 00:32:01.877: INFO: Waiting for pod pod-subpath-test-dynamicpv-gbfd to disappear
May 11 00:32:01.947: INFO: Pod pod-subpath-test-dynamicpv-gbfd still exists
May 11 00:32:03.877: INFO: Waiting for pod pod-subpath-test-dynamicpv-gbfd to disappear
... skipping 20 lines ...
May 11 00:32:20.552: INFO: At 2022-05-11 00:26:57 +0000 UTC - event for ebs.csi.aws.comh2hs2: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:32:20.552: INFO: At 2022-05-11 00:26:57 +0000 UTC - event for ebs.csi.aws.comh2hs2: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-4055/ebs.csi.aws.comh2hs2"
May 11 00:32:20.552: INFO: At 2022-05-11 00:26:57 +0000 UTC - event for ebs.csi.aws.comh2hs2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:32:20.552: INFO: At 2022-05-11 00:27:00 +0000 UTC - event for ebs.csi.aws.comh2hs2: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-16a887ee-fa4b-4064-80f5-2fca7b362d44
May 11 00:32:20.552: INFO: At 2022-05-11 00:27:01 +0000 UTC - event for pod-subpath-test-dynamicpv-gbfd: {default-scheduler } Scheduled: Successfully assigned provisioning-4055/pod-subpath-test-dynamicpv-gbfd to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:32:20.552: INFO: At 2022-05-11 00:27:03 +0000 UTC - event for pod-subpath-test-dynamicpv-gbfd: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-16a887ee-fa4b-4064-80f5-2fca7b362d44" 
May 11 00:32:20.552: INFO: At 2022-05-11 00:27:09 +0000 UTC - event for pod-subpath-test-dynamicpv-gbfd: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-16a887ee-fa4b-4064-80f5-2fca7b362d44" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-16a887ee-fa4b-4064-80f5-2fca7b362d44/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped)
May 11 00:32:20.552: INFO: At 2022-05-11 00:29:04 +0000 UTC - event for pod-subpath-test-dynamicpv-gbfd: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-lj678]: timed out waiting for the condition
May 11 00:32:20.618: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:32:20.618: INFO: 
May 11 00:32:20.728: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:32:20.794: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 5462 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:29:59 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:29:59 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:29:59 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:29:59 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 104 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support readOnly directory specified in the volumeMount [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:369

    Unexpected error:
        <*errors.errorString | 0xc0011712c0>: {
            s: "expected pod \"pod-subpath-test-dynamicpv-gbfd\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-gbfd\" to be \"Succeeded or Failed\"",
        }
        expected pod "pod-subpath-test-dynamicpv-gbfd" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-gbfd" to be "Succeeded or Failed"
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742
------------------------------
SS
------------------------------
... skipping 64 lines ...
May 11 00:33:13.059: INFO: At 2022-05-11 00:28:12 +0000 UTC - event for pvc-cf2w7: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:33:13.059: INFO: At 2022-05-11 00:28:12 +0000 UTC - event for pvc-cf2w7: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:33:13.059: INFO: At 2022-05-11 00:28:12 +0000 UTC - event for pvc-cf2w7: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-862/pvc-cf2w7"
May 11 00:33:13.059: INFO: At 2022-05-11 00:28:16 +0000 UTC - event for pod-6428baf1-8a97-42e3-8ebd-23873519a292: {default-scheduler } Scheduled: Successfully assigned provisioning-862/pod-6428baf1-8a97-42e3-8ebd-23873519a292 to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:33:13.059: INFO: At 2022-05-11 00:28:16 +0000 UTC - event for pvc-cf2w7: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-5f93625e-3db8-4ee1-b539-812d61179c45
May 11 00:33:13.059: INFO: At 2022-05-11 00:28:18 +0000 UTC - event for pod-6428baf1-8a97-42e3-8ebd-23873519a292: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-5f93625e-3db8-4ee1-b539-812d61179c45" 
May 11 00:33:13.059: INFO: At 2022-05-11 00:28:20 +0000 UTC - event for pod-6428baf1-8a97-42e3-8ebd-23873519a292: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-5f93625e-3db8-4ee1-b539-812d61179c45" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5f93625e-3db8-4ee1-b539-812d61179c45/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped)
May 11 00:33:13.059: INFO: At 2022-05-11 00:30:19 +0000 UTC - event for pod-6428baf1-8a97-42e3-8ebd-23873519a292: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-k4zc7]: timed out waiting for the condition
May 11 00:33:13.119: INFO: POD                                       NODE                                         PHASE    GRACE  CONDITIONS
May 11 00:33:13.119: INFO: pod-6428baf1-8a97-42e3-8ebd-23873519a292  ip-172-20-58-208.us-west-2.compute.internal  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:16 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:16 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:16 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:16 +0000 UTC  }]
May 11 00:33:13.119: INFO: 
May 11 00:33:13.180: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
... skipping 107 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] provisioning
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should provision storage with mount options [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:179

    Unexpected error:
        <*errors.errorString | 0xc00095cc00>: {
            s: "pod \"pod-6428baf1-8a97-42e3-8ebd-23873519a292\" is not Running: timed out waiting for the condition",
        }
        pod "pod-6428baf1-8a97-42e3-8ebd-23873519a292" is not Running: timed out waiting for the condition
    occurred

... skipping 39 lines ...
May 11 00:37:45.528: INFO: At 2022-05-11 00:32:23 +0000 UTC - event for ebs.csi.aws.comz58f6: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:37:45.528: INFO: At 2022-05-11 00:32:23 +0000 UTC - event for ebs.csi.aws.comz58f6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-1782/ebs.csi.aws.comz58f6"
May 11 00:37:45.528: INFO: At 2022-05-11 00:32:23 +0000 UTC - event for ebs.csi.aws.comz58f6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:37:45.528: INFO: At 2022-05-11 00:32:26 +0000 UTC - event for ebs.csi.aws.comz58f6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-dd22cb69-e292-4c49-b739-695dfa3de75d
May 11 00:37:45.528: INFO: At 2022-05-11 00:32:27 +0000 UTC - event for pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2: {default-scheduler } Scheduled: Successfully assigned volume-expand-1782/pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2 to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:37:45.528: INFO: At 2022-05-11 00:32:29 +0000 UTC - event for pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-dd22cb69-e292-4c49-b739-695dfa3de75d" 
May 11 00:37:45.528: INFO: At 2022-05-11 00:32:31 +0000 UTC - event for pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-dd22cb69-e292-4c49-b739-695dfa3de75d" : rpc error: code = Internal desc = Could not mount "/dev/nvme4n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-dd22cb69-e292-4c49-b739-695dfa3de75d/e3aa5ee0-d83b-4179-8c4b-0c2b2ca18ce7": mount failed: exec: "mount": executable file not found in $PATH
Mounting command: mount
Mounting arguments:  -o bind /dev/nvme4n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-dd22cb69-e292-4c49-b739-695dfa3de75d/e3aa5ee0-d83b-4179-8c4b-0c2b2ca18ce7
Output: 
May 11 00:37:45.528: INFO: At 2022-05-11 00:34:30 +0000 UTC - event for pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-bz859 volume1]: timed out waiting for the condition
May 11 00:37:45.597: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:37:45.597: INFO: 
... skipping 108 lines ...
  [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    Verify if offline PVC expansion works [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:174

    While creating pods for resizing
    Unexpected error:
        <*errors.errorString | 0xc000d26090>: {
            s: "pod \"pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2\" is not Running: timed out waiting for the condition",
        }
        pod "pod-db044970-c471-4fb2-b5cd-4af8ad7cf6a2" is not Running: timed out waiting for the condition
    occurred

... skipping 100 lines ...
May 11 00:22:49.258: INFO: Creating resource for dynamic PV
May 11 00:22:49.258: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass snapshotting-745-e2e-sckvt92
STEP: creating a claim
May 11 00:22:49.319: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: [init] starting a pod to use the claim
May 11 00:22:49.505: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-rvm4s" in namespace "snapshotting-745" to be "Succeeded or Failed"
May 11 00:22:49.565: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 60.029248ms
May 11 00:22:51.635: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 2.129770944s
May 11 00:22:53.696: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 4.191065408s
May 11 00:22:55.758: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 6.252870772s
May 11 00:22:57.819: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 8.314460168s
May 11 00:22:59.881: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 10.375897268s
... skipping 425 lines ...
May 11 00:37:38.809: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m49.303929217s
May 11 00:37:40.873: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m51.368271716s
May 11 00:37:42.938: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m53.432538161s
May 11 00:37:45.002: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m55.496678874s
May 11 00:37:47.066: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m57.561307372s
May 11 00:37:49.136: INFO: Pod "pvc-snapshottable-tester-rvm4s": Phase="Pending", Reason="", readiness=false. Elapsed: 14m59.631200364s
May 11 00:37:51.201: INFO: Error getting logs for pod pvc-snapshottable-tester-rvm4s: the server rejected our request for an unknown reason (get pods pvc-snapshottable-tester-rvm4s)
May 11 00:37:51.201: INFO: Deleting pod "pvc-snapshottable-tester-rvm4s" in namespace "snapshotting-745"
May 11 00:37:51.265: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-rvm4s" to be fully deleted
[AfterEach] volume snapshot controller
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:223
STEP: Deleting pvc
May 11 00:37:59.516: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comlzpmt"
... skipping 9 lines ...
May 11 00:38:09.905: INFO: At 2022-05-11 00:22:49 +0000 UTC - event for ebs.csi.aws.comlzpmt: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:38:09.905: INFO: At 2022-05-11 00:22:49 +0000 UTC - event for ebs.csi.aws.comlzpmt: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "snapshotting-745/ebs.csi.aws.comlzpmt"
May 11 00:38:09.905: INFO: At 2022-05-11 00:22:49 +0000 UTC - event for ebs.csi.aws.comlzpmt: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:38:09.905: INFO: At 2022-05-11 00:22:52 +0000 UTC - event for ebs.csi.aws.comlzpmt: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-f4383d4f-4ef3-48bb-9906-67bbb78e61a2
May 11 00:38:09.905: INFO: At 2022-05-11 00:22:53 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {default-scheduler } Scheduled: Successfully assigned snapshotting-745/pvc-snapshottable-tester-rvm4s to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:38:09.905: INFO: At 2022-05-11 00:22:57 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f4383d4f-4ef3-48bb-9906-67bbb78e61a2" 
May 11 00:38:09.905: INFO: At 2022-05-11 00:23:01 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-f4383d4f-4ef3-48bb-9906-67bbb78e61a2" : rpc error: code = Internal desc = could not format "/dev/nvme2n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f4383d4f-4ef3-48bb-9906-67bbb78e61a2/globalmount": failed to get disk format of disk /dev/nvme2n1: signal: segmentation fault (core dumped)
May 11 00:38:09.905: INFO: At 2022-05-11 00:24:56 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[my-volume default-token-9lzt8]: timed out waiting for the condition
May 11 00:38:09.905: INFO: At 2022-05-11 00:36:20 +0000 UTC - event for pvc-snapshottable-tester-rvm4s: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[default-token-9lzt8 my-volume]: timed out waiting for the condition
May 11 00:38:09.968: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:38:09.968: INFO: 
May 11 00:38:10.033: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
... skipping 109 lines ...
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108
       [BeforeEach]
      /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:227
        should check snapshot fields, check restore correctly works after modifying source data, check deletion
        /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:245

        Unexpected error:
            <*errors.errorString | 0xc000ec0500>: {
                s: "Gave up after waiting 15m0s for pod \"pvc-snapshottable-tester-rvm4s\" to be \"Succeeded or Failed\"",
            }
            Gave up after waiting 15m0s for pod "pvc-snapshottable-tester-rvm4s" to be "Succeeded or Failed"
        occurred

        /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:675
------------------------------
SS
------------------------------
... skipping 14 lines ...
May 11 00:28:16.879: INFO: Creating resource for dynamic PV
May 11 00:28:16.879: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-7648-e2e-scnwzwz
STEP: creating a claim
May 11 00:28:16.942: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod to format volume volume-prep-provisioning-7648
May 11 00:28:17.136: INFO: Waiting up to 5m0s for pod "volume-prep-provisioning-7648" in namespace "provisioning-7648" to be "Succeeded or Failed"
May 11 00:28:17.198: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 62.290659ms
May 11 00:28:19.261: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 2.125545261s
May 11 00:28:21.325: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 4.189537195s
May 11 00:28:23.389: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 6.253355519s
May 11 00:28:25.452: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 8.316860034s
May 11 00:28:27.516: INFO: Pod "volume-prep-provisioning-7648": Phase="Pending", Reason="", readiness=false. Elapsed: 10.380834754s
... skipping 210 lines ...
May 11 00:38:22.750: INFO: At 2022-05-11 00:28:17 +0000 UTC - event for ebs.csi.aws.coml86g9: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:38:22.750: INFO: At 2022-05-11 00:28:17 +0000 UTC - event for ebs.csi.aws.coml86g9: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-7648/ebs.csi.aws.coml86g9"
May 11 00:38:22.750: INFO: At 2022-05-11 00:28:17 +0000 UTC - event for ebs.csi.aws.coml86g9: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:38:22.750: INFO: At 2022-05-11 00:28:20 +0000 UTC - event for ebs.csi.aws.coml86g9: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-84a89b77-3de7-41cf-aa28-e12be2c5d992
May 11 00:38:22.750: INFO: At 2022-05-11 00:28:21 +0000 UTC - event for volume-prep-provisioning-7648: {default-scheduler } Scheduled: Successfully assigned provisioning-7648/volume-prep-provisioning-7648 to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:38:22.750: INFO: At 2022-05-11 00:28:23 +0000 UTC - event for volume-prep-provisioning-7648: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-84a89b77-3de7-41cf-aa28-e12be2c5d992" 
May 11 00:38:22.750: INFO: At 2022-05-11 00:28:37 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-84a89b77-3de7-41cf-aa28-e12be2c5d992" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-84a89b77-3de7-41cf-aa28-e12be2c5d992/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped)
May 11 00:38:22.750: INFO: At 2022-05-11 00:30:24 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume default-token-82s79]: timed out waiting for the condition
May 11 00:38:22.750: INFO: At 2022-05-11 00:32:42 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[default-token-82s79 test-volume]: timed out waiting for the condition
May 11 00:38:22.751: INFO: At 2022-05-11 00:35:13 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[default-token-82s79 test-volume]: error processing PVC provisioning-7648/ebs.csi.aws.coml86g9: PVC is being deleted
May 11 00:38:22.751: INFO: At 2022-05-11 00:35:25 +0000 UTC - event for volume-prep-provisioning-7648: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume default-token-82s79]: error processing PVC provisioning-7648/ebs.csi.aws.coml86g9: PVC is being deleted
May 11 00:38:22.812: INFO: POD                            NODE                                         PHASE    GRACE  CONDITIONS
May 11 00:38:22.812: INFO: volume-prep-provisioning-7648  ip-172-20-58-208.us-west-2.compute.internal  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:21 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:21 +0000 UTC ContainersNotReady containers with unready status: [init-volume-provisioning-7648]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:21 +0000 UTC ContainersNotReady containers with unready status: [init-volume-provisioning-7648]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:28:21 +0000 UTC  }]
May 11 00:38:22.812: INFO: 
May 11 00:38:22.875: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:38:22.997: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 6539 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 107 lines ...
  [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should verify container cannot write to subpath readonly volumes [Slow] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:427

    while waiting for volume init pod to succeed
    Unexpected error:
        <*errors.errorString | 0xc000e7b970>: {
            s: "Gave up after waiting 5m0s for pod \"volume-prep-provisioning-7648\" to be \"Succeeded or Failed\"",
        }
        Gave up after waiting 5m0s for pod "volume-prep-provisioning-7648" to be "Succeeded or Failed"
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:1021
------------------------------
S
------------------------------
... skipping 15 lines ...
May 11 00:33:15.421: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volume-718-e2e-sc9fjj4
STEP: creating a claim
May 11 00:33:15.482: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod exec-volume-test-dynamicpv-zknn
STEP: Creating a pod to test exec-volume-test
May 11 00:33:15.667: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-zknn" in namespace "volume-718" to be "Succeeded or Failed"
May 11 00:33:15.726: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 59.449473ms
May 11 00:33:17.787: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 2.120179768s
May 11 00:33:19.848: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4.181218643s
May 11 00:33:21.909: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 6.242141011s
May 11 00:33:23.970: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 8.303205829s
May 11 00:33:26.032: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 10.365303911s
... skipping 134 lines ...
May 11 00:38:04.294: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m48.627324704s
May 11 00:38:06.356: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m50.689019173s
May 11 00:38:08.417: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m52.750081875s
May 11 00:38:10.478: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m54.810943904s
May 11 00:38:12.538: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m56.870734141s
May 11 00:38:14.599: INFO: Pod "exec-volume-test-dynamicpv-zknn": Phase="Pending", Reason="", readiness=false. Elapsed: 4m58.93234705s
May 11 00:38:16.772: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "exec-volume-test-dynamicpv-zknn" container "exec-container-dynamicpv-zknn": the server rejected our request for an unknown reason (get pods exec-volume-test-dynamicpv-zknn)
STEP: delete the pod
May 11 00:38:16.834: INFO: Waiting for pod exec-volume-test-dynamicpv-zknn to disappear
May 11 00:38:16.903: INFO: Pod exec-volume-test-dynamicpv-zknn still exists
May 11 00:38:18.903: INFO: Waiting for pod exec-volume-test-dynamicpv-zknn to disappear
May 11 00:38:18.964: INFO: Pod exec-volume-test-dynamicpv-zknn still exists
May 11 00:38:20.903: INFO: Waiting for pod exec-volume-test-dynamicpv-zknn to disappear
... skipping 21 lines ...
May 11 00:38:44.511: INFO: At 2022-05-11 00:33:15 +0000 UTC - event for ebs.csi.aws.com4p82g: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:38:44.511: INFO: At 2022-05-11 00:33:15 +0000 UTC - event for ebs.csi.aws.com4p82g: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-718/ebs.csi.aws.com4p82g"
May 11 00:38:44.511: INFO: At 2022-05-11 00:33:15 +0000 UTC - event for ebs.csi.aws.com4p82g: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:38:44.511: INFO: At 2022-05-11 00:33:18 +0000 UTC - event for ebs.csi.aws.com4p82g: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-f0894353-965d-4327-8e02-4fdafbce2a94
May 11 00:38:44.511: INFO: At 2022-05-11 00:33:19 +0000 UTC - event for exec-volume-test-dynamicpv-zknn: {default-scheduler } Scheduled: Successfully assigned volume-718/exec-volume-test-dynamicpv-zknn to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:38:44.511: INFO: At 2022-05-11 00:33:21 +0000 UTC - event for exec-volume-test-dynamicpv-zknn: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f0894353-965d-4327-8e02-4fdafbce2a94" 
May 11 00:38:44.511: INFO: At 2022-05-11 00:33:23 +0000 UTC - event for exec-volume-test-dynamicpv-zknn: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-f0894353-965d-4327-8e02-4fdafbce2a94" : rpc error: code = Internal desc = could not format "/dev/nvme5n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f0894353-965d-4327-8e02-4fdafbce2a94/globalmount": failed to get disk format of disk /dev/nvme5n1: signal: segmentation fault (core dumped)
May 11 00:38:44.511: INFO: At 2022-05-11 00:35:22 +0000 UTC - event for exec-volume-test-dynamicpv-zknn: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[vol1], unattached volumes=[default-token-c7xvv vol1]: timed out waiting for the condition
May 11 00:38:44.570: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:38:44.570: INFO: 
May 11 00:38:44.630: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:38:44.690: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 6539 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:35:01 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 104 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:196

    Unexpected error:
        <*errors.errorString | 0xc0009bc730>: {
            s: "expected pod \"exec-volume-test-dynamicpv-zknn\" success: Gave up after waiting 5m0s for pod \"exec-volume-test-dynamicpv-zknn\" to be \"Succeeded or Failed\"",
        }
        expected pod "exec-volume-test-dynamicpv-zknn" success: Gave up after waiting 5m0s for pod "exec-volume-test-dynamicpv-zknn" to be "Succeeded or Failed"
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742
------------------------------
S
------------------------------
... skipping 56 lines ...
May 11 00:42:54.839: INFO: Waiting for pod external-injector to disappear
May 11 00:42:54.907: INFO: Pod external-injector still exists
May 11 00:42:56.840: INFO: Waiting for pod external-injector to disappear
May 11 00:42:56.907: INFO: Pod external-injector still exists
May 11 00:42:58.839: INFO: Waiting for pod external-injector to disappear
May 11 00:42:58.906: INFO: Pod external-injector no longer exists
May 11 00:42:58.906: FAIL: Failed to create injector pod: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.prepareSnapshotDataSourceForProvisioning(_, {{0xc000668588, 0x11}, {0x281b564, 0x8}, {0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...)
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:845 +0x2e5
k8s.io/kubernetes/test/e2e/storage/testsuites.(*provisioningTestSuite).DefineTests.func4()
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:219 +0x565
... skipping 10 lines ...
May 11 00:42:58.974: INFO: At 2022-05-11 00:37:48 +0000 UTC - event for pvc-7jv5w: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:42:58.974: INFO: At 2022-05-11 00:37:48 +0000 UTC - event for pvc-7jv5w: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:42:58.974: INFO: At 2022-05-11 00:37:48 +0000 UTC - event for pvc-7jv5w: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-9780/pvc-7jv5w"
May 11 00:42:58.974: INFO: At 2022-05-11 00:37:51 +0000 UTC - event for pvc-7jv5w: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326
May 11 00:42:58.974: INFO: At 2022-05-11 00:37:52 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned provisioning-9780/external-injector to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:42:58.974: INFO: At 2022-05-11 00:37:54 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326" 
May 11 00:42:58.974: INFO: At 2022-05-11 00:37:56 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326" : rpc error: code = Internal desc = Could not mount "/dev/nvme1n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326/2f3a4d6d-b61b-4760-b803-4c161a00c99e": mount failed: exec: "mount": executable file not found in $PATH
Mounting command: mount
Mounting arguments:  -o bind /dev/nvme1n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-d494cfc4-bbf1-4100-9a06-932d98dd3326/2f3a4d6d-b61b-4760-b803-4c161a00c99e
Output: 
May 11 00:42:58.974: INFO: At 2022-05-11 00:39:55 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[default-token-g5btb external-volume-0]: timed out waiting for the condition
May 11 00:42:59.041: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:42:59.041: INFO: 
... skipping 108 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (block volmode)] provisioning
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:200

    May 11 00:42:58.906: Failed to create injector pod: timed out waiting for the condition

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:845
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
May 11 00:43:01.225: INFO: Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping
... skipping 137 lines ...
May 11 00:43:34.667: INFO: At 2022-05-11 00:38:12 +0000 UTC - event for ebs.csi.aws.com9xp9m: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:43:34.667: INFO: At 2022-05-11 00:38:12 +0000 UTC - event for ebs.csi.aws.com9xp9m: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-449/ebs.csi.aws.com9xp9m"
May 11 00:43:34.667: INFO: At 2022-05-11 00:38:12 +0000 UTC - event for ebs.csi.aws.com9xp9m: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:43:34.668: INFO: At 2022-05-11 00:38:15 +0000 UTC - event for ebs.csi.aws.com9xp9m: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5
May 11 00:43:34.668: INFO: At 2022-05-11 00:38:16 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {default-scheduler } Scheduled: Successfully assigned volume-expand-449/pod-988c7646-bdd1-4d20-8724-748b64f2ac31 to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:43:34.668: INFO: At 2022-05-11 00:38:18 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5" 
May 11 00:43:34.668: INFO: At 2022-05-11 00:38:24 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5" : rpc error: code = Internal desc = Could not mount "/dev/nvme2n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5/7f2a9b3e-2c9f-4cf4-86ac-f9ae4d40d19c": mount failed: exec: "mount": executable file not found in $PATH
Mounting command: mount
Mounting arguments:  -o bind /dev/nvme2n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-e4041bb0-7977-415c-bf4a-27e5ee48f6b5/7f2a9b3e-2c9f-4cf4-86ac-f9ae4d40d19c
Output: 
May 11 00:43:34.668: INFO: At 2022-05-11 00:40:19 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-bqs8c]: timed out waiting for the condition
May 11 00:43:34.668: INFO: At 2022-05-11 00:42:35 +0000 UTC - event for pod-988c7646-bdd1-4d20-8724-748b64f2ac31: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-bqs8c volume1]: timed out waiting for the condition
May 11 00:43:34.731: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
... skipping 110 lines ...
  [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should resize volume when PVC is edited while pod is using it [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246

    While creating pods for resizing
    Unexpected error:
        <*errors.errorString | 0xc000e84850>: {
            s: "pod \"pod-988c7646-bdd1-4d20-8724-748b64f2ac31\" is not Running: timed out waiting for the condition",
        }
        pod "pod-988c7646-bdd1-4d20-8724-748b64f2ac31" is not Running: timed out waiting for the condition
    occurred

... skipping 77 lines ...

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
S
------------------------------
External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (default fs)] subPath 
  should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240

[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185
STEP: Creating a kubernetes client
May 11 00:38:46.971: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig
STEP: Building a namespace api object, basename provisioning
STEP: Waiting for a default service account to be provisioned in namespace
[It] should fail if subpath directory is outside the volume [Slow][LinuxOnly]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240
May 11 00:38:47.270: INFO: Creating resource for dynamic PV
May 11 00:38:47.270: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-6601-e2e-sc4rktk
STEP: creating a claim
May 11 00:38:47.330: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod pod-subpath-test-dynamicpv-hc2l
STEP: Checking for subpath error in container status
May 11 00:43:47.692: INFO: Deleting pod "pod-subpath-test-dynamicpv-hc2l" in namespace "provisioning-6601"
May 11 00:43:47.755: INFO: Wait up to 5m0s for pod "pod-subpath-test-dynamicpv-hc2l" to be fully deleted
STEP: Deleting pod
May 11 00:43:59.877: INFO: Deleting pod "pod-subpath-test-dynamicpv-hc2l" in namespace "provisioning-6601"
STEP: Deleting pvc
May 11 00:44:00.057: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comrdr6x"
... skipping 10 lines ...
May 11 00:44:15.484: INFO: At 2022-05-11 00:38:47 +0000 UTC - event for ebs.csi.aws.comrdr6x: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:44:15.484: INFO: At 2022-05-11 00:38:47 +0000 UTC - event for ebs.csi.aws.comrdr6x: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-6601/ebs.csi.aws.comrdr6x"
May 11 00:44:15.484: INFO: At 2022-05-11 00:38:47 +0000 UTC - event for ebs.csi.aws.comrdr6x: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:44:15.484: INFO: At 2022-05-11 00:38:50 +0000 UTC - event for ebs.csi.aws.comrdr6x: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-a421b3a2-148a-4428-af8c-d4db1354169f
May 11 00:44:15.484: INFO: At 2022-05-11 00:38:51 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {default-scheduler } Scheduled: Successfully assigned provisioning-6601/pod-subpath-test-dynamicpv-hc2l to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:44:15.484: INFO: At 2022-05-11 00:38:53 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-a421b3a2-148a-4428-af8c-d4db1354169f" 
May 11 00:44:15.484: INFO: At 2022-05-11 00:38:55 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-a421b3a2-148a-4428-af8c-d4db1354169f" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a421b3a2-148a-4428-af8c-d4db1354169f/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped)
May 11 00:44:15.484: INFO: At 2022-05-11 00:40:54 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[default-token-rkmdb test-volume liveness-probe-volume]: timed out waiting for the condition
May 11 00:44:15.484: INFO: At 2022-05-11 00:43:10 +0000 UTC - event for pod-subpath-test-dynamicpv-hc2l: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-rkmdb]: timed out waiting for the condition
May 11 00:44:15.544: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:44:15.544: INFO: 
May 11 00:44:15.605: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
... skipping 102 lines ...

• Failure [330.577 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240

    while waiting for subpath failure
    Unexpected error:
        <*errors.errorString | 0xc000df8690>: {
            s: "error waiting for pod subpath error to occur: timed out waiting for the condition",
        }
        error waiting for pod subpath error to occur: timed out waiting for the condition
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:718
------------------------------
SS
------------------------------
... skipping 73 lines ...
May 11 00:48:25.868: INFO: At 2022-05-11 00:43:01 +0000 UTC - event for ebs.csi.aws.comd7qq8: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:48:25.868: INFO: At 2022-05-11 00:43:01 +0000 UTC - event for ebs.csi.aws.comd7qq8: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "multivolume-4341/ebs.csi.aws.comd7qq8"
May 11 00:48:25.868: INFO: At 2022-05-11 00:43:01 +0000 UTC - event for ebs.csi.aws.comd7qq8: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:48:25.868: INFO: At 2022-05-11 00:43:05 +0000 UTC - event for ebs.csi.aws.comd7qq8: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-19813fee-5743-4aab-aaca-c7531eb29322
May 11 00:48:25.868: INFO: At 2022-05-11 00:43:05 +0000 UTC - event for pod-10936628-816b-4642-99b9-62fcb80b1a42: {default-scheduler } Scheduled: Successfully assigned multivolume-4341/pod-10936628-816b-4642-99b9-62fcb80b1a42 to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:48:25.868: INFO: At 2022-05-11 00:43:09 +0000 UTC - event for pod-10936628-816b-4642-99b9-62fcb80b1a42: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-19813fee-5743-4aab-aaca-c7531eb29322" 
May 11 00:48:25.868: INFO: At 2022-05-11 00:43:21 +0000 UTC - event for pod-10936628-816b-4642-99b9-62fcb80b1a42: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-19813fee-5743-4aab-aaca-c7531eb29322" : rpc error: code = Internal desc = Could not mount "/dev/nvme1n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-19813fee-5743-4aab-aaca-c7531eb29322/62728771-9321-49bf-97fb-1d037bcc797b": mount failed: exec: "mount": executable file not found in $PATH
Mounting command: mount
Mounting arguments:  -o bind /dev/nvme1n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-19813fee-5743-4aab-aaca-c7531eb29322/62728771-9321-49bf-97fb-1d037bcc797b
Output: 
May 11 00:48:25.868: INFO: At 2022-05-11 00:45:08 +0000 UTC - event for pod-10936628-816b-4642-99b9-62fcb80b1a42: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-l22vh volume1]: timed out waiting for the condition
May 11 00:48:25.931: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:48:25.931: INFO: 
... skipping 107 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should concurrently access the single read-only volume from pods on the same node [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:337

    Unexpected error:
        <*errors.errorString | 0xc000650c00>: {
            s: "pod \"pod-10936628-816b-4642-99b9-62fcb80b1a42\" is not Running: timed out waiting for the condition",
        }
        pod "pod-10936628-816b-4642-99b9-62fcb80b1a42" is not Running: timed out waiting for the condition
    occurred

... skipping 67 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Pre-provisioned PV (block volmode)] volumeMode
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297

    Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
... skipping 31 lines ...
May 11 00:49:26.419: INFO: Waiting for pod external-injector to disappear
May 11 00:49:26.482: INFO: Pod external-injector still exists
May 11 00:49:28.421: INFO: Waiting for pod external-injector to disappear
May 11 00:49:28.484: INFO: Pod external-injector still exists
May 11 00:49:30.419: INFO: Waiting for pod external-injector to disappear
May 11 00:49:30.482: INFO: Pod external-injector no longer exists
May 11 00:49:30.482: FAIL: Failed to create injector pod: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3()
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 +0x485
github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes.TestEBSCSI(0x0?)
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes/e2e_test.go:82 +0x311
... skipping 17 lines ...
May 11 00:49:46.055: INFO: At 2022-05-11 00:44:17 +0000 UTC - event for ebs.csi.aws.comvbr2r: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:49:46.055: INFO: At 2022-05-11 00:44:18 +0000 UTC - event for ebs.csi.aws.comvbr2r: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-5268/ebs.csi.aws.comvbr2r"
May 11 00:49:46.055: INFO: At 2022-05-11 00:44:18 +0000 UTC - event for ebs.csi.aws.comvbr2r: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:49:46.055: INFO: At 2022-05-11 00:44:21 +0000 UTC - event for ebs.csi.aws.comvbr2r: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-75405376-72c7-4c02-ad51-fc2ac9604439
May 11 00:49:46.055: INFO: At 2022-05-11 00:44:22 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned volume-5268/external-injector to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:49:46.055: INFO: At 2022-05-11 00:44:24 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-75405376-72c7-4c02-ad51-fc2ac9604439" 
May 11 00:49:46.055: INFO: At 2022-05-11 00:44:25 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-75405376-72c7-4c02-ad51-fc2ac9604439" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-75405376-72c7-4c02-ad51-fc2ac9604439/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped)
May 11 00:49:46.055: INFO: At 2022-05-11 00:46:25 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[default-token-j8p46 external-volume-0]: timed out waiting for the condition
May 11 00:49:46.055: INFO: At 2022-05-11 00:48:39 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 default-token-j8p46]: timed out waiting for the condition
May 11 00:49:46.117: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:49:46.117: INFO: 
May 11 00:49:46.180: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
... skipping 105 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (xfs)][Slow] volumes
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should store data [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159

    May 11 00:49:30.482: Failed to create injector pod: timed out waiting for the condition

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186
------------------------------
SS
------------------------------
[BeforeEach] [Testpattern: Inline-volume (default fs)] subPath
... skipping 127 lines ...
May 11 00:53:41.753: INFO: At 2022-05-11 00:43:37 +0000 UTC - event for ebs.csi.aws.comspczh: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:53:41.753: INFO: At 2022-05-11 00:43:37 +0000 UTC - event for ebs.csi.aws.comspczh: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-368/ebs.csi.aws.comspczh"
May 11 00:53:41.753: INFO: At 2022-05-11 00:43:37 +0000 UTC - event for ebs.csi.aws.comspczh: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:53:41.753: INFO: At 2022-05-11 00:43:40 +0000 UTC - event for ebs.csi.aws.comspczh: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-91290b1c-6974-4222-9ba7-b492c4852d5c
May 11 00:53:41.753: INFO: At 2022-05-11 00:43:41 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-368/pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:53:41.753: INFO: At 2022-05-11 00:43:43 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-91290b1c-6974-4222-9ba7-b492c4852d5c" 
May 11 00:53:41.753: INFO: At 2022-05-11 00:43:45 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-91290b1c-6974-4222-9ba7-b492c4852d5c" : rpc error: code = Internal desc = could not format "/dev/nvme2n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-91290b1c-6974-4222-9ba7-b492c4852d5c/globalmount": failed to get disk format of disk /dev/nvme2n1: signal: segmentation fault (core dumped)
May 11 00:53:41.753: INFO: At 2022-05-11 00:45:44 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-25hd6]: timed out waiting for the condition
May 11 00:53:41.753: INFO: At 2022-05-11 00:50:30 +0000 UTC - event for pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-25hd6]: error processing PVC fsgroupchangepolicy-368/ebs.csi.aws.comspczh: PVC is being deleted
May 11 00:53:41.815: INFO: POD                                       NODE                                         PHASE    GRACE  CONDITIONS
May 11 00:53:41.815: INFO: pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc  ip-172-20-58-208.us-west-2.compute.internal  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:43:41 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:43:41 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:43:41 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:43:41 +0000 UTC  }]
May 11 00:53:41.815: INFO: 
May 11 00:53:41.877: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:53:41.940: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 9915 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 104 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208

    Unexpected error:
        <*errors.errorString | 0xc000e859f0>: {
            s: "pod \"pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc\" is not Running: timed out waiting for the condition",
        }
        pod "pod-876ecaa7-fabc-4840-a33c-8fdd95455cfc" is not Running: timed out waiting for the condition
    occurred

... skipping 129 lines ...

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:121
------------------------------
SS
------------------------------
External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (block volmode)] volumeMode 
  should fail to use a volume in a pod with mismatched mode [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297

[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (block volmode)] volumeMode
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185
STEP: Creating a kubernetes client
May 11 00:53:44.385: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig
STEP: Building a namespace api object, basename volumemode
STEP: Waiting for a default service account to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297
May 11 00:53:44.693: INFO: Creating resource for dynamic PV
May 11 00:53:44.693: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volumemode-1189-e2e-sc7gm6c
STEP: creating a claim
STEP: Creating pod
STEP: Waiting for the pod to fail
May 11 00:53:51.129: INFO: Deleting pod "pod-73bb5975-8d3a-4977-96c5-df68dbddfaa5" in namespace "volumemode-1189"
May 11 00:53:51.193: INFO: Wait up to 5m0s for pod "pod-73bb5975-8d3a-4977-96c5-df68dbddfaa5" to be fully deleted
STEP: Deleting pvc
May 11 00:53:59.441: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.com6nrgn"
May 11 00:53:59.503: INFO: Waiting up to 5m0s for PersistentVolume pvc-23a6249b-ea7b-41b1-bddf-ecd415806f51 to get deleted
May 11 00:53:59.564: INFO: PersistentVolume pvc-23a6249b-ea7b-41b1-bddf-ecd415806f51 found and phase=Released (60.899036ms)
... skipping 7 lines ...

• [SLOW TEST:20.432 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (block volmode)] volumeMode
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297
------------------------------
SSSS
------------------------------
External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller  
  should check snapshot fields, check restore correctly works after modifying source data, check deletion
... skipping 12 lines ...
May 11 00:38:25.238: INFO: Creating resource for dynamic PV
May 11 00:38:25.238: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass snapshotting-7107-e2e-scchsnn
STEP: creating a claim
May 11 00:38:25.301: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: [init] starting a pod to use the claim
May 11 00:38:25.496: INFO: Waiting up to 15m0s for pod "pvc-snapshottable-tester-dnhvk" in namespace "snapshotting-7107" to be "Succeeded or Failed"
May 11 00:38:25.558: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 61.838627ms
May 11 00:38:27.621: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 2.124265578s
May 11 00:38:29.684: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 4.187645262s
May 11 00:38:31.748: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 6.25117358s
May 11 00:38:33.810: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 8.313391461s
May 11 00:38:35.874: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 10.377360624s
... skipping 424 lines ...
May 11 00:53:13.365: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m47.869022719s
May 11 00:53:15.431: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m49.934972559s
May 11 00:53:17.496: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m51.99973686s
May 11 00:53:19.562: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m54.065730536s
May 11 00:53:21.629: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m56.1327446s
May 11 00:53:23.694: INFO: Pod "pvc-snapshottable-tester-dnhvk": Phase="Pending", Reason="", readiness=false. Elapsed: 14m58.197557875s
May 11 00:53:25.766: INFO: Error getting logs for pod pvc-snapshottable-tester-dnhvk: the server rejected our request for an unknown reason (get pods pvc-snapshottable-tester-dnhvk)
May 11 00:53:25.766: INFO: Deleting pod "pvc-snapshottable-tester-dnhvk" in namespace "snapshotting-7107"
May 11 00:53:25.832: INFO: Wait up to 5m0s for pod "pvc-snapshottable-tester-dnhvk" to be fully deleted
[AfterEach] volume snapshot controller
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:223
STEP: Deleting pvc
May 11 00:53:30.092: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comhmmcd"
... skipping 14 lines ...
May 11 00:54:05.844: INFO: At 2022-05-11 00:38:25 +0000 UTC - event for ebs.csi.aws.comhmmcd: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:54:05.844: INFO: At 2022-05-11 00:38:25 +0000 UTC - event for ebs.csi.aws.comhmmcd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "snapshotting-7107/ebs.csi.aws.comhmmcd"
May 11 00:54:05.844: INFO: At 2022-05-11 00:38:25 +0000 UTC - event for ebs.csi.aws.comhmmcd: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:54:05.844: INFO: At 2022-05-11 00:38:28 +0000 UTC - event for ebs.csi.aws.comhmmcd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-41b1404e-2a2b-4bf8-af71-80141d1608f4
May 11 00:54:05.844: INFO: At 2022-05-11 00:38:29 +0000 UTC - event for pvc-snapshottable-tester-dnhvk: {default-scheduler } Scheduled: Successfully assigned snapshotting-7107/pvc-snapshottable-tester-dnhvk to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:54:05.844: INFO: At 2022-05-11 00:38:31 +0000 UTC - event for pvc-snapshottable-tester-dnhvk: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-41b1404e-2a2b-4bf8-af71-80141d1608f4" 
May 11 00:54:05.844: INFO: At 2022-05-11 00:38:33 +0000 UTC - event for pvc-snapshottable-tester-dnhvk: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-41b1404e-2a2b-4bf8-af71-80141d1608f4" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-41b1404e-2a2b-4bf8-af71-80141d1608f4/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped)
May 11 00:54:05.844: INFO: At 2022-05-11 00:40:32 +0000 UTC - event for pvc-snapshottable-tester-dnhvk: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[my-volume], unattached volumes=[my-volume default-token-jjfwc]: timed out waiting for the condition
May 11 00:54:05.908: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:54:05.908: INFO: 
May 11 00:54:05.973: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:54:06.039: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 9915 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:50:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 106 lines ...
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:108
       [BeforeEach]
      /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:227
        should check snapshot fields, check restore correctly works after modifying source data, check deletion
        /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/snapshottable.go:245

        Unexpected error:
            <*errors.errorString | 0xc000e7a9d0>: {
                s: "Gave up after waiting 15m0s for pod \"pvc-snapshottable-tester-dnhvk\" to be \"Succeeded or Failed\"",
            }
            Gave up after waiting 15m0s for pod "pvc-snapshottable-tester-dnhvk" to be "Succeeded or Failed"
        occurred

        /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:675
------------------------------
S
------------------------------
... skipping 6 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Inline-volume (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:256

    Driver "ebs.csi.aws.com" does not support volume type "InlineVolume" - skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
... skipping 29 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Inline-volume (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if subpath with backstepping is outside the volume [Slow][LinuxOnly] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:278

    Driver "ebs.csi.aws.com" does not support volume type "InlineVolume" - skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
... skipping 64 lines ...

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/multivolume.go:372
------------------------------
SS
------------------------------
External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode 
  should fail to use a volume in a pod with mismatched mode [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297

[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185
STEP: Creating a kubernetes client
May 11 00:54:08.562: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig
STEP: Building a namespace api object, basename volumemode
STEP: Waiting for a default service account to be provisioned in namespace
[It] should fail to use a volume in a pod with mismatched mode [Slow]
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297
May 11 00:54:08.887: INFO: Creating resource for dynamic PV
May 11 00:54:08.887: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volumemode-3181-e2e-scmv558
STEP: creating a claim
STEP: Creating pod
STEP: Waiting for the pod to fail
May 11 00:54:15.357: INFO: Deleting pod "pod-4a8173af-146c-41bf-a656-b727c763696b" in namespace "volumemode-3181"
May 11 00:54:15.422: INFO: Wait up to 5m0s for pod "pod-4a8173af-146c-41bf-a656-b727c763696b" to be fully deleted
STEP: Deleting pvc
May 11 00:54:19.681: INFO: Deleting PersistentVolumeClaim "ebs.csi.aws.comkx6zt"
May 11 00:54:19.747: INFO: Waiting up to 5m0s for PersistentVolume pvc-8f11720b-90e3-43cb-b257-00a5b5358822 to get deleted
May 11 00:54:19.811: INFO: PersistentVolume pvc-8f11720b-90e3-43cb-b257-00a5b5358822 found and phase=Released (64.328985ms)
... skipping 7 lines ...

• [SLOW TEST:16.512 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (filesystem volmode)] volumeMode
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
May 11 00:54:25.075: INFO: Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping
[AfterEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
... skipping 2 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if subpath file is outside the volume [Slow][LinuxOnly] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:256

    Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
External Storage [Driver: ebs.csi.aws.com] [Testpattern: Dynamic PV (immediate binding)] topology 
  should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192

[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:51
[BeforeEach] [Testpattern: Dynamic PV (immediate binding)] topology
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:185
STEP: Creating a kubernetes client
May 11 00:54:25.077: INFO: >>> kubeConfig: /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/hack/e2e/csi-test-artifacts/test-cluster-27998.k8s.local.kops.kubeconfig
STEP: Building a namespace api object, basename topology
STEP: Waiting for a default service account to be provisioned in namespace
[It] should fail to schedule a pod which has topologies that conflict with AllowedTopologies
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:192
May 11 00:54:25.464: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2c]
May 11 00:54:25.465: INFO: found topology map[topology.ebs.csi.aws.com/zone:us-west-2a]
May 11 00:54:25.465: INFO: Creating storage class object and pvc object for driver - sc: &StorageClass{ObjectMeta:{topology-8346-e2e-sct7vcr      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Provisioner:ebs.csi.aws.com,Parameters:map[string]string{},ReclaimPolicy:nil,MountOptions:[],AllowVolumeExpansion:nil,VolumeBindingMode:*Immediate,AllowedTopologies:[]TopologySelectorTerm{{[{topology.ebs.csi.aws.com/zone [us-west-2c]}]},},}, pvc: &PersistentVolumeClaim{ObjectMeta:{ pvc- topology-8346    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1073741824 0} {<nil>} 1Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*topology-8346-e2e-sct7vcr,VolumeMode:nil,DataSource:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},},}
STEP: Creating sc
STEP: Creating pvc
... skipping 148 lines ...
May 11 00:55:35.021: INFO: At 2022-05-11 00:49:48 +0000 UTC - event for pvc-jfqn6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:55:35.021: INFO: At 2022-05-11 00:49:48 +0000 UTC - event for pvc-jfqn6: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:55:35.021: INFO: At 2022-05-11 00:49:48 +0000 UTC - event for pvc-jfqn6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "topology-5464/pvc-jfqn6"
May 11 00:55:35.021: INFO: At 2022-05-11 00:49:52 +0000 UTC - event for pvc-jfqn6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-cd897812-8156-49c9-bc2a-fe57fd7b6ff1
May 11 00:55:35.021: INFO: At 2022-05-11 00:49:53 +0000 UTC - event for pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed: {default-scheduler } Scheduled: Successfully assigned topology-5464/pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:55:35.021: INFO: At 2022-05-11 00:49:55 +0000 UTC - event for pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-cd897812-8156-49c9-bc2a-fe57fd7b6ff1" 
May 11 00:55:35.021: INFO: At 2022-05-11 00:49:57 +0000 UTC - event for pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-cd897812-8156-49c9-bc2a-fe57fd7b6ff1" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-cd897812-8156-49c9-bc2a-fe57fd7b6ff1/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped)
May 11 00:55:35.021: INFO: At 2022-05-11 00:51:56 +0000 UTC - event for pod-c574ed1c-08dc-42fc-9cbb-4b481644e2ed: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-kknk7]: timed out waiting for the condition
May 11 00:55:35.083: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:55:35.083: INFO: 
May 11 00:55:35.146: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:55:35.208: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 11143 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 104 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (delayed binding)] topology
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should provision a volume and schedule a pod with AllowedTopologies [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:164

    Unexpected error:
        <*errors.errorString | 0xc0001b0200>: {
            s: "timed out waiting for the condition",
        }
        timed out waiting for the condition
    occurred

... skipping 90 lines ...
May 11 00:58:33.116: INFO: At 2022-05-11 00:48:28 +0000 UTC - event for ebs.csi.aws.comvstp8: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 00:58:33.116: INFO: At 2022-05-11 00:48:28 +0000 UTC - event for ebs.csi.aws.comvstp8: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-3578/ebs.csi.aws.comvstp8"
May 11 00:58:33.116: INFO: At 2022-05-11 00:48:28 +0000 UTC - event for ebs.csi.aws.comvstp8: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:58:33.116: INFO: At 2022-05-11 00:48:31 +0000 UTC - event for ebs.csi.aws.comvstp8: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-f896ae32-7a0d-47f3-b1fc-ecdc5f856453
May 11 00:58:33.116: INFO: At 2022-05-11 00:48:32 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-3578/pod-54bcd33a-ebbb-444d-b919-3b04b75461cb to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:58:33.116: INFO: At 2022-05-11 00:48:34 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-f896ae32-7a0d-47f3-b1fc-ecdc5f856453" 
May 11 00:58:33.116: INFO: At 2022-05-11 00:48:40 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-f896ae32-7a0d-47f3-b1fc-ecdc5f856453" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-f896ae32-7a0d-47f3-b1fc-ecdc5f856453/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped)
May 11 00:58:33.116: INFO: At 2022-05-11 00:50:35 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-cv7gc]: timed out waiting for the condition
May 11 00:58:33.116: INFO: At 2022-05-11 00:55:21 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-cv7gc]: error processing PVC fsgroupchangepolicy-3578/ebs.csi.aws.comvstp8: PVC is being deleted
May 11 00:58:33.116: INFO: At 2022-05-11 00:55:34 +0000 UTC - event for pod-54bcd33a-ebbb-444d-b919-3b04b75461cb: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-cv7gc volume1]: error processing PVC fsgroupchangepolicy-3578/ebs.csi.aws.comvstp8: PVC is being deleted
May 11 00:58:33.182: INFO: POD                                       NODE                                         PHASE    GRACE  CONDITIONS
May 11 00:58:33.182: INFO: pod-54bcd33a-ebbb-444d-b919-3b04b75461cb  ip-172-20-58-208.us-west-2.compute.internal  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:48:32 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:48:32 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:48:32 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:48:32 +0000 UTC  }]
May 11 00:58:33.182: INFO: 
May 11 00:58:33.250: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 00:58:33.315: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 11143 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 00:55:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 108 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208

    Unexpected error:
        <*errors.errorString | 0xc000d7cc10>: {
            s: "pod \"pod-54bcd33a-ebbb-444d-b919-3b04b75461cb\" is not Running: timed out waiting for the condition",
        }
        pod "pod-54bcd33a-ebbb-444d-b919-3b04b75461cb" is not Running: timed out waiting for the condition
    occurred

... skipping 59 lines ...
May 11 00:59:20.332: INFO: At 2022-05-11 00:54:05 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {default-scheduler } FailedScheduling: 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
May 11 00:59:20.332: INFO: At 2022-05-11 00:54:05 +0000 UTC - event for pvc-h7zwd: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 00:59:20.332: INFO: At 2022-05-11 00:54:05 +0000 UTC - event for pvc-h7zwd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "topology-1172/pvc-h7zwd"
May 11 00:59:20.332: INFO: At 2022-05-11 00:54:08 +0000 UTC - event for pvc-h7zwd: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-b3bb2cad-3ece-433e-af63-5356213b8d90
May 11 00:59:20.332: INFO: At 2022-05-11 00:54:13 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {default-scheduler } Scheduled: Successfully assigned topology-1172/pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d to ip-172-20-58-208.us-west-2.compute.internal
May 11 00:59:20.332: INFO: At 2022-05-11 00:54:15 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-b3bb2cad-3ece-433e-af63-5356213b8d90" 
May 11 00:59:20.332: INFO: At 2022-05-11 00:54:21 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-b3bb2cad-3ece-433e-af63-5356213b8d90" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b3bb2cad-3ece-433e-af63-5356213b8d90/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped)
May 11 00:59:20.332: INFO: At 2022-05-11 00:56:16 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-gj8gj]: timed out waiting for the condition
May 11 00:59:20.332: INFO: At 2022-05-11 00:58:31 +0000 UTC - event for pod-fd9e2e57-5554-4cf3-9b3a-02b92cc7391d: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-gj8gj volume1]: timed out waiting for the condition
May 11 00:59:20.396: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 00:59:20.396: INFO: 
May 11 00:59:20.460: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
... skipping 107 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (immediate binding)] topology
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should provision a volume and schedule a pod with AllowedTopologies [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/topology.go:164

    Unexpected error:
        <*errors.errorString | 0xc00021c1d0>: {
            s: "timed out waiting for the condition",
        }
        timed out waiting for the condition
    occurred

... skipping 19 lines ...
May 11 00:55:37.484: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-9086-e2e-scdp2cz
STEP: creating a claim
May 11 00:55:37.547: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod pod-subpath-test-dynamicpv-hcll
STEP: Creating a pod to test subpath
May 11 00:55:37.737: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-hcll" in namespace "provisioning-9086" to be "Succeeded or Failed"
May 11 00:55:37.799: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 62.124875ms
May 11 00:55:39.867: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 2.12960971s
May 11 00:55:41.930: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4.192522117s
May 11 00:55:43.993: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 6.255893518s
May 11 00:55:46.056: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 8.319279374s
May 11 00:55:48.120: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 10.3824466s
... skipping 134 lines ...
May 11 01:00:26.764: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.02633169s
May 11 01:00:28.828: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.090290015s
May 11 01:00:30.891: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.153644063s
May 11 01:00:32.954: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.216772742s
May 11 01:00:35.018: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.280438904s
May 11 01:00:37.082: INFO: Pod "pod-subpath-test-dynamicpv-hcll": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.344416449s
May 11 01:00:39.218: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-hcll" container "init-volume-dynamicpv-hcll": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-hcll)
May 11 01:00:39.280: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-hcll" container "test-init-volume-dynamicpv-hcll": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-hcll)
May 11 01:00:39.342: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-hcll" container "test-container-subpath-dynamicpv-hcll": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-hcll)
STEP: delete the pod
May 11 01:00:39.404: INFO: Waiting for pod pod-subpath-test-dynamicpv-hcll to disappear
May 11 01:00:39.464: INFO: Pod pod-subpath-test-dynamicpv-hcll still exists
May 11 01:00:41.466: INFO: Waiting for pod pod-subpath-test-dynamicpv-hcll to disappear
May 11 01:00:41.527: INFO: Pod pod-subpath-test-dynamicpv-hcll still exists
May 11 01:00:43.465: INFO: Waiting for pod pod-subpath-test-dynamicpv-hcll to disappear
... skipping 21 lines ...
May 11 01:01:05.146: INFO: At 2022-05-11 00:55:37 +0000 UTC - event for ebs.csi.aws.com2mqtx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 01:01:05.146: INFO: At 2022-05-11 00:55:37 +0000 UTC - event for ebs.csi.aws.com2mqtx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-9086/ebs.csi.aws.com2mqtx"
May 11 01:01:05.146: INFO: At 2022-05-11 00:55:37 +0000 UTC - event for ebs.csi.aws.com2mqtx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 01:01:05.146: INFO: At 2022-05-11 00:55:41 +0000 UTC - event for ebs.csi.aws.com2mqtx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-29de2a7d-293f-472b-8f18-15e80949cd22
May 11 01:01:05.146: INFO: At 2022-05-11 00:55:41 +0000 UTC - event for pod-subpath-test-dynamicpv-hcll: {default-scheduler } Scheduled: Successfully assigned provisioning-9086/pod-subpath-test-dynamicpv-hcll to ip-172-20-58-208.us-west-2.compute.internal
May 11 01:01:05.146: INFO: At 2022-05-11 00:55:43 +0000 UTC - event for pod-subpath-test-dynamicpv-hcll: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-29de2a7d-293f-472b-8f18-15e80949cd22" 
May 11 01:01:05.146: INFO: At 2022-05-11 00:55:49 +0000 UTC - event for pod-subpath-test-dynamicpv-hcll: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-29de2a7d-293f-472b-8f18-15e80949cd22" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-29de2a7d-293f-472b-8f18-15e80949cd22/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped)
May 11 01:01:05.146: INFO: At 2022-05-11 00:57:44 +0000 UTC - event for pod-subpath-test-dynamicpv-hcll: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-dczjf]: timed out waiting for the condition
May 11 01:01:05.206: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 01:01:05.206: INFO: 
May 11 01:01:05.272: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 01:01:05.332: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 12255 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 105 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support readOnly file specified in the volumeMount [LinuxOnly] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:384

    Unexpected error:
        <*errors.errorString | 0xc0007ca6b0>: {
            s: "expected pod \"pod-subpath-test-dynamicpv-hcll\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-hcll\" to be \"Succeeded or Failed\"",
        }
        expected pod "pod-subpath-test-dynamicpv-hcll" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-hcll" to be "Succeeded or Failed"
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742
------------------------------
S
------------------------------
... skipping 134 lines ...
May 11 01:03:36.269: INFO: Waiting for pod external-injector to disappear
May 11 01:03:36.332: INFO: Pod external-injector still exists
May 11 01:03:38.333: INFO: Waiting for pod external-injector to disappear
May 11 01:03:38.396: INFO: Pod external-injector still exists
May 11 01:03:40.332: INFO: Waiting for pod external-injector to disappear
May 11 01:03:40.395: INFO: Pod external-injector no longer exists
May 11 01:03:40.395: FAIL: Failed to create injector pod: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.prepareSnapshotDataSourceForProvisioning(_, {{0xc000ca45e8, 0x11}, {0x281b564, 0x8}, {0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...)
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:845 +0x2e5
k8s.io/kubernetes/test/e2e/storage/testsuites.(*provisioningTestSuite).DefineTests.func4()
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:219 +0x565
... skipping 10 lines ...
May 11 01:03:40.460: INFO: At 2022-05-11 00:58:35 +0000 UTC - event for pvc-v262f: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 01:03:40.460: INFO: At 2022-05-11 00:58:36 +0000 UTC - event for pvc-v262f: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 01:03:40.460: INFO: At 2022-05-11 00:58:36 +0000 UTC - event for pvc-v262f: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-8211/pvc-v262f"
May 11 01:03:40.460: INFO: At 2022-05-11 00:58:39 +0000 UTC - event for pvc-v262f: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-972585f9-1635-4b28-9ebb-eed239d81d95
May 11 01:03:40.460: INFO: At 2022-05-11 00:58:40 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned provisioning-8211/external-injector to ip-172-20-58-208.us-west-2.compute.internal
May 11 01:03:40.460: INFO: At 2022-05-11 00:58:42 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-972585f9-1635-4b28-9ebb-eed239d81d95" 
May 11 01:03:40.460: INFO: At 2022-05-11 00:58:55 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-972585f9-1635-4b28-9ebb-eed239d81d95" : rpc error: code = Internal desc = could not format "/dev/nvme5n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-972585f9-1635-4b28-9ebb-eed239d81d95/globalmount": failed to get disk format of disk /dev/nvme5n1: signal: segmentation fault (core dumped)
May 11 01:03:40.460: INFO: At 2022-05-11 01:00:43 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[external-volume-0 default-token-wptcn]: timed out waiting for the condition
May 11 01:03:40.522: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 01:03:40.522: INFO: 
May 11 01:03:40.586: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 01:03:40.649: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 12255 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 105 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] provisioning
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:200

    May 11 01:03:40.395: Failed to create injector pod: timed out waiting for the condition

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/provisioning.go:845
------------------------------
S
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
... skipping 150 lines ...
May 11 01:04:31.839: INFO: At 2022-05-11 00:54:27 +0000 UTC - event for ebs.csi.aws.com2m7jx: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 01:04:31.839: INFO: At 2022-05-11 00:54:27 +0000 UTC - event for ebs.csi.aws.com2m7jx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-5351/ebs.csi.aws.com2m7jx"
May 11 01:04:31.839: INFO: At 2022-05-11 00:54:27 +0000 UTC - event for ebs.csi.aws.com2m7jx: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 01:04:31.839: INFO: At 2022-05-11 00:54:30 +0000 UTC - event for ebs.csi.aws.com2m7jx: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-4c7a1ae8-cc68-4821-b0c5-fcf206df0b14
May 11 01:04:31.839: INFO: At 2022-05-11 00:54:31 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-5351/pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce to ip-172-20-58-208.us-west-2.compute.internal
May 11 01:04:31.839: INFO: At 2022-05-11 00:54:33 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-4c7a1ae8-cc68-4821-b0c5-fcf206df0b14" 
May 11 01:04:31.839: INFO: At 2022-05-11 00:54:39 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-4c7a1ae8-cc68-4821-b0c5-fcf206df0b14" : rpc error: code = Internal desc = could not format "/dev/nvme2n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-4c7a1ae8-cc68-4821-b0c5-fcf206df0b14/globalmount": failed to get disk format of disk /dev/nvme2n1: signal: segmentation fault (core dumped)
May 11 01:04:31.839: INFO: At 2022-05-11 00:56:34 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-b85sh]: timed out waiting for the condition
May 11 01:04:31.839: INFO: At 2022-05-11 01:01:23 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-b85sh]: error processing PVC fsgroupchangepolicy-5351/ebs.csi.aws.com2m7jx: PVC is being deleted
May 11 01:04:31.839: INFO: At 2022-05-11 01:03:09 +0000 UTC - event for pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-b85sh volume1]: error processing PVC fsgroupchangepolicy-5351/ebs.csi.aws.com2m7jx: PVC is being deleted
May 11 01:04:31.903: INFO: POD                                       NODE                                         PHASE    GRACE  CONDITIONS
May 11 01:04:31.903: INFO: pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce  ip-172-20-58-208.us-west-2.compute.internal  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:54:31 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:54:31 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:54:31 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 00:54:31 +0000 UTC  }]
May 11 01:04:31.903: INFO: 
May 11 01:04:32.019: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 01:04:32.086: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 12255 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 104 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208

    Unexpected error:
        <*errors.errorString | 0xc000bef4b0>: {
            s: "pod \"pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce\" is not Running: timed out waiting for the condition",
        }
        pod "pod-fed1f598-0ae8-4419-9b70-c8c4c3e63bce" is not Running: timed out waiting for the condition
    occurred

... skipping 19 lines ...
May 11 00:59:22.770: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass provisioning-5676-e2e-sc4r66s
STEP: creating a claim
May 11 00:59:22.834: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod pod-subpath-test-dynamicpv-8sr8
STEP: Creating a pod to test subpath
May 11 00:59:23.027: INFO: Waiting up to 5m0s for pod "pod-subpath-test-dynamicpv-8sr8" in namespace "provisioning-5676" to be "Succeeded or Failed"
May 11 00:59:23.090: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 62.997809ms
May 11 00:59:25.154: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 2.126870771s
May 11 00:59:27.217: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4.190496891s
May 11 00:59:29.281: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 6.25422595s
May 11 00:59:31.345: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 8.318077861s
May 11 00:59:33.409: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 10.382322582s
... skipping 134 lines ...
May 11 01:04:12.192: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.164678583s
May 11 01:04:14.255: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.227695792s
May 11 01:04:16.319: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.291799386s
May 11 01:04:18.383: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.355717571s
May 11 01:04:20.446: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.41924009s
May 11 01:04:22.509: INFO: Pod "pod-subpath-test-dynamicpv-8sr8": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.482176849s
May 11 01:04:24.639: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-8sr8" container "init-volume-dynamicpv-8sr8": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-8sr8)
May 11 01:04:24.711: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "pod-subpath-test-dynamicpv-8sr8" container "test-container-subpath-dynamicpv-8sr8": the server rejected our request for an unknown reason (get pods pod-subpath-test-dynamicpv-8sr8)
STEP: delete the pod
May 11 01:04:24.775: INFO: Waiting for pod pod-subpath-test-dynamicpv-8sr8 to disappear
May 11 01:04:24.838: INFO: Pod pod-subpath-test-dynamicpv-8sr8 still exists
May 11 01:04:26.839: INFO: Waiting for pod pod-subpath-test-dynamicpv-8sr8 to disappear
May 11 01:04:26.905: INFO: Pod pod-subpath-test-dynamicpv-8sr8 still exists
May 11 01:04:28.838: INFO: Waiting for pod pod-subpath-test-dynamicpv-8sr8 to disappear
... skipping 15 lines ...
May 11 01:04:44.546: INFO: At 2022-05-11 00:59:22 +0000 UTC - event for ebs.csi.aws.comqszv6: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 01:04:44.546: INFO: At 2022-05-11 00:59:23 +0000 UTC - event for ebs.csi.aws.comqszv6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "provisioning-5676/ebs.csi.aws.comqszv6"
May 11 01:04:44.546: INFO: At 2022-05-11 00:59:23 +0000 UTC - event for ebs.csi.aws.comqszv6: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 01:04:44.546: INFO: At 2022-05-11 00:59:26 +0000 UTC - event for ebs.csi.aws.comqszv6: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-08fa6a3f-6bf5-4e29-949d-7b759b03166a
May 11 01:04:44.546: INFO: At 2022-05-11 00:59:27 +0000 UTC - event for pod-subpath-test-dynamicpv-8sr8: {default-scheduler } Scheduled: Successfully assigned provisioning-5676/pod-subpath-test-dynamicpv-8sr8 to ip-172-20-58-208.us-west-2.compute.internal
May 11 01:04:44.546: INFO: At 2022-05-11 00:59:28 +0000 UTC - event for pod-subpath-test-dynamicpv-8sr8: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-08fa6a3f-6bf5-4e29-949d-7b759b03166a" 
May 11 01:04:44.546: INFO: At 2022-05-11 00:59:30 +0000 UTC - event for pod-subpath-test-dynamicpv-8sr8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-08fa6a3f-6bf5-4e29-949d-7b759b03166a" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-08fa6a3f-6bf5-4e29-949d-7b759b03166a/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped)
May 11 01:04:44.546: INFO: At 2022-05-11 01:01:30 +0000 UTC - event for pod-subpath-test-dynamicpv-8sr8: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[test-volume], unattached volumes=[test-volume liveness-probe-volume default-token-ptx8z]: timed out waiting for the condition
May 11 01:04:44.609: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 01:04:44.609: INFO: 
May 11 01:04:44.672: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 01:04:44.735: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 12255 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:00:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 106 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should support existing single file [LinuxOnly] [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:219

    Unexpected error:
        <*errors.errorString | 0xc000ab4f50>: {
            s: "expected pod \"pod-subpath-test-dynamicpv-8sr8\" success: Gave up after waiting 5m0s for pod \"pod-subpath-test-dynamicpv-8sr8\" to be \"Succeeded or Failed\"",
        }
        expected pod "pod-subpath-test-dynamicpv-8sr8" success: Gave up after waiting 5m0s for pod "pod-subpath-test-dynamicpv-8sr8" to be "Succeeded or Failed"
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742
------------------------------
SSS
------------------------------
... skipping 46 lines ...
May 11 01:04:34.346: INFO: Using claimSize:1Gi, test suite supported size:{ 1Mi}, driver(ebs.csi.aws.com) supported size:{ 1Mi} 
STEP: creating a StorageClass volume-7154-e2e-scrts84
STEP: creating a claim
May 11 01:04:34.411: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
STEP: Creating pod exec-volume-test-dynamicpv-llsv
STEP: Creating a pod to test exec-volume-test
May 11 01:04:34.608: INFO: Waiting up to 5m0s for pod "exec-volume-test-dynamicpv-llsv" in namespace "volume-7154" to be "Succeeded or Failed"
May 11 01:04:34.673: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 64.429394ms
May 11 01:04:36.737: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 2.129275792s
May 11 01:04:38.802: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4.194094111s
May 11 01:04:40.868: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 6.259542802s
May 11 01:04:42.933: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 8.325115844s
May 11 01:04:44.999: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 10.390464897s
... skipping 134 lines ...
May 11 01:09:23.856: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m49.247705716s
May 11 01:09:25.920: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m51.312227721s
May 11 01:09:27.986: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m53.37754408s
May 11 01:09:30.052: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m55.4435775s
May 11 01:09:32.117: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m57.509306208s
May 11 01:09:34.182: INFO: Pod "exec-volume-test-dynamicpv-llsv": Phase="Pending", Reason="", readiness=false. Elapsed: 4m59.573550902s
May 11 01:09:36.327: INFO: Failed to get logs from node "ip-172-20-58-208.us-west-2.compute.internal" pod "exec-volume-test-dynamicpv-llsv" container "exec-container-dynamicpv-llsv": the server rejected our request for an unknown reason (get pods exec-volume-test-dynamicpv-llsv)
STEP: delete the pod
May 11 01:09:36.394: INFO: Waiting for pod exec-volume-test-dynamicpv-llsv to disappear
May 11 01:09:36.459: INFO: Pod exec-volume-test-dynamicpv-llsv still exists
May 11 01:09:38.460: INFO: Waiting for pod exec-volume-test-dynamicpv-llsv to disappear
May 11 01:09:38.524: INFO: Pod exec-volume-test-dynamicpv-llsv still exists
May 11 01:09:40.459: INFO: Waiting for pod exec-volume-test-dynamicpv-llsv to disappear
... skipping 13 lines ...
May 11 01:09:56.110: INFO: At 2022-05-11 01:04:34 +0000 UTC - event for ebs.csi.aws.comkq4km: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 01:09:56.110: INFO: At 2022-05-11 01:04:34 +0000 UTC - event for ebs.csi.aws.comkq4km: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-7154/ebs.csi.aws.comkq4km"
May 11 01:09:56.110: INFO: At 2022-05-11 01:04:34 +0000 UTC - event for ebs.csi.aws.comkq4km: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 01:09:56.110: INFO: At 2022-05-11 01:04:37 +0000 UTC - event for ebs.csi.aws.comkq4km: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-bc3bdf1d-ea10-43dd-a13f-0704ea409c92
May 11 01:09:56.110: INFO: At 2022-05-11 01:04:38 +0000 UTC - event for exec-volume-test-dynamicpv-llsv: {default-scheduler } Scheduled: Successfully assigned volume-7154/exec-volume-test-dynamicpv-llsv to ip-172-20-58-208.us-west-2.compute.internal
May 11 01:09:56.110: INFO: At 2022-05-11 01:04:40 +0000 UTC - event for exec-volume-test-dynamicpv-llsv: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-bc3bdf1d-ea10-43dd-a13f-0704ea409c92" 
May 11 01:09:56.111: INFO: At 2022-05-11 01:04:42 +0000 UTC - event for exec-volume-test-dynamicpv-llsv: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-bc3bdf1d-ea10-43dd-a13f-0704ea409c92" : rpc error: code = Internal desc = could not format "/dev/nvme1n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-bc3bdf1d-ea10-43dd-a13f-0704ea409c92/globalmount": failed to get disk format of disk /dev/nvme1n1: signal: segmentation fault (core dumped)
May 11 01:09:56.111: INFO: At 2022-05-11 01:06:41 +0000 UTC - event for exec-volume-test-dynamicpv-llsv: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[vol1], unattached volumes=[vol1 default-token-2zzh7]: timed out waiting for the condition
May 11 01:09:56.174: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 01:09:56.174: INFO: 
May 11 01:09:56.239: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 01:09:56.302: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 13431 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:05:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:05:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:05:06 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:05:06 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 104 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] volumes
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should allow exec of files on the volume [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:196

    Unexpected error:
        <*errors.errorString | 0xc00062f1e0>: {
            s: "expected pod \"exec-volume-test-dynamicpv-llsv\" success: Gave up after waiting 5m0s for pod \"exec-volume-test-dynamicpv-llsv\" to be \"Succeeded or Failed\"",
        }
        expected pod "exec-volume-test-dynamicpv-llsv" success: Gave up after waiting 5m0s for pod "exec-volume-test-dynamicpv-llsv" to be "Succeeded or Failed"
    occurred

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:742
------------------------------
SS
------------------------------
... skipping 6 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Pre-provisioned PV (default fs)] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267

    Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
... skipping 37 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail to use a volume in a pod with mismatched mode [Slow] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumemode.go:297

    Driver "ebs.csi.aws.com" does not support volume type "PreprovisionedPV" - skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:263
------------------------------
... skipping 27 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if non-existent subpath is outside the volume [Slow][LinuxOnly] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:267

    Driver ebs.csi.aws.com doesn't support ntfs -- skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:121
------------------------------
... skipping 91 lines ...
May 11 01:10:15.753: INFO: At 2022-05-11 01:04:47 +0000 UTC - event for ebs.csi.aws.comhm6kv: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 01:10:15.753: INFO: At 2022-05-11 01:04:47 +0000 UTC - event for ebs.csi.aws.comhm6kv: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-expand-4949/ebs.csi.aws.comhm6kv"
May 11 01:10:15.753: INFO: At 2022-05-11 01:04:47 +0000 UTC - event for ebs.csi.aws.comhm6kv: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 01:10:15.753: INFO: At 2022-05-11 01:04:51 +0000 UTC - event for ebs.csi.aws.comhm6kv: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-8a9b18e2-2fdb-42f8-8097-4f86f8076203
May 11 01:10:15.753: INFO: At 2022-05-11 01:04:51 +0000 UTC - event for pod-47dd5244-65cc-4afc-8cb7-141bcd1257af: {default-scheduler } Scheduled: Successfully assigned volume-expand-4949/pod-47dd5244-65cc-4afc-8cb7-141bcd1257af to ip-172-20-58-208.us-west-2.compute.internal
May 11 01:10:15.753: INFO: At 2022-05-11 01:04:53 +0000 UTC - event for pod-47dd5244-65cc-4afc-8cb7-141bcd1257af: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-8a9b18e2-2fdb-42f8-8097-4f86f8076203" 
May 11 01:10:15.753: INFO: At 2022-05-11 01:04:55 +0000 UTC - event for pod-47dd5244-65cc-4afc-8cb7-141bcd1257af: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-8a9b18e2-2fdb-42f8-8097-4f86f8076203" : rpc error: code = Internal desc = could not format "/dev/nvme5n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8a9b18e2-2fdb-42f8-8097-4f86f8076203/globalmount": failed to get disk format of disk /dev/nvme5n1: signal: segmentation fault (core dumped)
May 11 01:10:15.753: INFO: At 2022-05-11 01:06:54 +0000 UTC - event for pod-47dd5244-65cc-4afc-8cb7-141bcd1257af: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-jb4mz]: timed out waiting for the condition
May 11 01:10:15.813: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 01:10:15.813: INFO: 
May 11 01:10:15.879: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 01:10:15.940: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 14492 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 105 lines ...
  [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should resize volume when PVC is edited while pod is using it [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volume_expand.go:246

    While creating pods for resizing
    Unexpected error:
        <*errors.errorString | 0xc000a4ac10>: {
            s: "pod \"pod-47dd5244-65cc-4afc-8cb7-141bcd1257af\" is not Running: timed out waiting for the condition",
        }
        pod "pod-47dd5244-65cc-4afc-8cb7-141bcd1257af" is not Running: timed out waiting for the condition
    occurred

... skipping 142 lines ...
May 11 01:11:12.657: INFO: At 2022-05-11 01:01:08 +0000 UTC - event for ebs.csi.aws.comhfh5f: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 01:11:12.657: INFO: At 2022-05-11 01:01:08 +0000 UTC - event for ebs.csi.aws.comhfh5f: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-438/ebs.csi.aws.comhfh5f"
May 11 01:11:12.657: INFO: At 2022-05-11 01:01:08 +0000 UTC - event for ebs.csi.aws.comhfh5f: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 01:11:12.657: INFO: At 2022-05-11 01:01:11 +0000 UTC - event for ebs.csi.aws.comhfh5f: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-806dd228-2c66-4ddf-ab12-d14bd6664724
May 11 01:11:12.657: INFO: At 2022-05-11 01:01:12 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-438/pod-55a64f41-8696-48c2-90b2-8c8067b04c60 to ip-172-20-58-208.us-west-2.compute.internal
May 11 01:11:12.657: INFO: At 2022-05-11 01:01:14 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-806dd228-2c66-4ddf-ab12-d14bd6664724" 
May 11 01:11:12.657: INFO: At 2022-05-11 01:01:28 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-806dd228-2c66-4ddf-ab12-d14bd6664724" : rpc error: code = Internal desc = could not format "/dev/nvme3n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-806dd228-2c66-4ddf-ab12-d14bd6664724/globalmount": failed to get disk format of disk /dev/nvme3n1: signal: segmentation fault (core dumped)
May 11 01:11:12.657: INFO: At 2022-05-11 01:03:15 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-jw2jl]: timed out waiting for the condition
May 11 01:11:12.657: INFO: At 2022-05-11 01:05:33 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-jw2jl volume1]: timed out waiting for the condition
May 11 01:11:12.657: INFO: At 2022-05-11 01:08:04 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-jw2jl]: error processing PVC fsgroupchangepolicy-438/ebs.csi.aws.comhfh5f: PVC is being deleted
May 11 01:11:12.657: INFO: At 2022-05-11 01:08:18 +0000 UTC - event for pod-55a64f41-8696-48c2-90b2-8c8067b04c60: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-jw2jl volume1]: error processing PVC fsgroupchangepolicy-438/ebs.csi.aws.comhfh5f: PVC is being deleted
May 11 01:11:12.717: INFO: POD                                       NODE                                         PHASE    GRACE  CONDITIONS
May 11 01:11:12.717: INFO: pod-55a64f41-8696-48c2-90b2-8c8067b04c60  ip-172-20-58-208.us-west-2.compute.internal  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:01:12 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:01:12 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:01:12 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:01:12 +0000 UTC  }]
May 11 01:11:12.717: INFO: 
May 11 01:11:12.780: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 01:11:12.840: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 14492 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 106 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208

    Unexpected error:
        <*errors.errorString | 0xc0001f9520>: {
            s: "pod \"pod-55a64f41-8696-48c2-90b2-8c8067b04c60\" is not Running: timed out waiting for the condition",
        }
        pod "pod-55a64f41-8696-48c2-90b2-8c8067b04c60" is not Running: timed out waiting for the condition
    occurred

... skipping 92 lines ...
May 11 01:13:47.640: INFO: At 2022-05-11 01:03:43 +0000 UTC - event for ebs.csi.aws.comh9v8t: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 01:13:47.640: INFO: At 2022-05-11 01:03:43 +0000 UTC - event for ebs.csi.aws.comh9v8t: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "fsgroupchangepolicy-5568/ebs.csi.aws.comh9v8t"
May 11 01:13:47.640: INFO: At 2022-05-11 01:03:43 +0000 UTC - event for ebs.csi.aws.comh9v8t: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 01:13:47.640: INFO: At 2022-05-11 01:03:46 +0000 UTC - event for ebs.csi.aws.comh9v8t: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-3ba11ba1-e9d0-4c83-8574-d5f5d3b3fbb8
May 11 01:13:47.640: INFO: At 2022-05-11 01:03:47 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {default-scheduler } Scheduled: Successfully assigned fsgroupchangepolicy-5568/pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34 to ip-172-20-58-208.us-west-2.compute.internal
May 11 01:13:47.640: INFO: At 2022-05-11 01:03:49 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-3ba11ba1-e9d0-4c83-8574-d5f5d3b3fbb8" 
May 11 01:13:47.640: INFO: At 2022-05-11 01:03:55 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: MountVolume.MountDevice failed for volume "pvc-3ba11ba1-e9d0-4c83-8574-d5f5d3b3fbb8" : rpc error: code = Internal desc = could not format "/dev/nvme4n1" and mount it at "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3ba11ba1-e9d0-4c83-8574-d5f5d3b3fbb8/globalmount": failed to get disk format of disk /dev/nvme4n1: signal: segmentation fault (core dumped)
May 11 01:13:47.640: INFO: At 2022-05-11 01:05:50 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-rsmtb]: timed out waiting for the condition
May 11 01:13:47.640: INFO: At 2022-05-11 01:10:37 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-rsmtb]: error processing PVC fsgroupchangepolicy-5568/ebs.csi.aws.comh9v8t: PVC is being deleted
May 11 01:13:47.640: INFO: At 2022-05-11 01:11:40 +0000 UTC - event for pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[default-token-rsmtb volume1]: error processing PVC fsgroupchangepolicy-5568/ebs.csi.aws.comh9v8t: PVC is being deleted
May 11 01:13:47.705: INFO: POD                                       NODE                                         PHASE    GRACE  CONDITIONS
May 11 01:13:47.705: INFO: pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34  ip-172-20-58-208.us-west-2.compute.internal  Pending         [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:03:47 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:03:47 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:03:47 +0000 UTC ContainersNotReady containers with unready status: [write-pod]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-05-11 01:03:47 +0000 UTC  }]
May 11 01:13:47.705: INFO: 
May 11 01:13:47.770: INFO: 
Logging node info for node ip-172-20-109-66.us-west-2.compute.internal
May 11 01:13:47.835: INFO: Node Info: &Node{ObjectMeta:{ip-172-20-109-66.us-west-2.compute.internal    47975c71-52c5-4a1c-b17e-c2a836d06284 14492 0 2022-05-11 00:08:56 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:c5.large beta.kubernetes.io/os:linux failure-domain.beta.kubernetes.io/region:us-west-2 failure-domain.beta.kubernetes.io/zone:us-west-2c kops.k8s.io/instancegroup:nodes-us-west-2c kubernetes.io/arch:amd64 kubernetes.io/hostname:ip-172-20-109-66.us-west-2.compute.internal kubernetes.io/os:linux kubernetes.io/role:node node-role.kubernetes.io/node: node.kubernetes.io/instance-type:c5.large topology.ebs.csi.aws.com/zone:us-west-2c topology.kubernetes.io/region:us-west-2 topology.kubernetes.io/zone:us-west-2c] map[csi.volume.kubernetes.io/nodeid:{"ebs.csi.aws.com":"i-042bf187445039118"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kops-controller Update v1 2022-05-11 00:08:56 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:kops.k8s.io/instancegroup":{},"f:kubernetes.io/role":{},"f:node-role.kubernetes.io/node":{}}}}} {kube-controller-manager Update v1 2022-05-11 00:08:58 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"100.96.1.0/24\"":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}} {kubelet Update v1 2022-05-11 00:09:42 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:csi.volume.kubernetes.io/nodeid":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.ebs.csi.aws.com/zone":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}},"f:status":{"f:addresses":{".":{},"k:{\"type\":\"ExternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"ExternalIP\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"Hostname\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalDNS\"}":{".":{},"f:address":{},"f:type":{}},"k:{\"type\":\"InternalIP\"}":{".":{},"f:address":{},"f:type":{}}},"f:allocatable":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:capacity":{".":{},"f:attachable-volumes-aws-ebs":{},"f:cpu":{},"f:ephemeral-storage":{},"f:hugepages-1Gi":{},"f:hugepages-2Mi":{},"f:memory":{},"f:pods":{}},"f:conditions":{".":{},"k:{\"type\":\"DiskPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"MemoryPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PIDPressure\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:daemonEndpoints":{"f:kubeletEndpoint":{"f:Port":{}}},"f:images":{},"f:nodeInfo":{"f:architecture":{},"f:bootID":{},"f:containerRuntimeVersion":{},"f:kernelVersion":{},"f:kubeProxyVersion":{},"f:kubeletVersion":{},"f:machineID":{},"f:operatingSystem":{},"f:osImage":{},"f:systemUUID":{}}}}}]},Spec:NodeSpec{PodCIDR:100.96.1.0/24,DoNotUseExternalID:,ProviderID:aws:///us-west-2c/i-042bf187445039118,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[100.96.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{133167038464 0} {<nil>}  BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3894337536 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-aws-ebs: {{25 0} {<nil>} 25 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{119850334420 0} {<nil>} 119850334420 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3789479936 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2022-05-11 00:08:58 +0000 UTC,LastTransitionTime:2022-05-11 00:08:58 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:08:56 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2022-05-11 01:10:07 +0000 UTC,LastTransitionTime:2022-05-11 00:09:06 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:172.20.109.66,},NodeAddress{Type:ExternalIP,Address:34.217.119.53,},NodeAddress{Type:Hostname,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:InternalDNS,Address:ip-172-20-109-66.us-west-2.compute.internal,},NodeAddress{Type:ExternalDNS,Address:ec2-34-217-119-53.us-west-2.compute.amazonaws.com,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:ec2899c215357bd399ecf9f2397c99da,SystemUUID:ec2899c2-1535-7bd3-99ec-f9f2397c99da,BootID:dd25b7ea-a79f-47f7-90db-9b9a320aa011,KernelVersion:5.13.0-1021-aws,OSImage:Ubuntu 20.04.4 LTS,ContainerRuntimeVersion:containerd://1.4.6,KubeletVersion:v1.20.8,KubeProxyVersion:v1.20.8,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-proxy@sha256:0c867c82a0a8ce6d093595f9d2e4b10517d6c9c26323075de9d82d9f7d056bfc k8s.gcr.io/kube-proxy:v1.20.8],SizeBytes:52056682,},ContainerImage{Names:[607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver@sha256:fb523044ca3b13e07c118f8e30db5cf7be0473f7131c8fecc04eb9806d351f60 607362164682.dkr.ecr.us-west-2.amazonaws.com/aws-ebs-csi-driver:27998],SizeBytes:26066331,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:695505fcfcc69f1cf35665dce487aad447adbb9af69b796d6437f869015d1157 k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1],SizeBytes:21212251,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:50c3cfd458fc8e0bf3c8c521eac39172009382fc66dc5044a330d137c6ed0b09 k8s.gcr.io/sig-storage/csi-attacher:v3.1.0],SizeBytes:20103959,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:9af9bf28430b00a0cedeb2ec29acadce45e6afcecd8bdf31c793c624cfa75fa7 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3],SizeBytes:19500777,},ContainerImage{Names:[k8s.gcr.io/cpa/cluster-proportional-autoscaler@sha256:67640771ad9fc56f109d5b01e020f0c858e7c890bb0eb15ba0ebd325df3285e7 k8s.gcr.io/cpa/cluster-proportional-autoscaler:1.8.3],SizeBytes:15191740,},ContainerImage{Names:[docker.io/coredns/coredns@sha256:642ff9910da6ea9a8624b0234eef52af9ca75ecbec474c5507cb096bdfbae4e5 docker.io/coredns/coredns:1.8.3],SizeBytes:12893350,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:529be2c9770add0cdd0c989115222ea9fc1be430c11095eb9f6dafcf98a36e2b k8s.gcr.io/sig-storage/livenessprobe:v2.4.0],SizeBytes:7960518,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:927d98197ec1141a368550822d18fa1c60bdae27b78b0c004f705f548c07814f k8s.gcr.io/pause:3.2],SizeBytes:299513,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
... skipping 106 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/fsgroupchangepolicy.go:208

    Unexpected error:
        <*errors.errorString | 0xc000ac0a10>: {
            s: "pod \"pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34\" is not Running: timed out waiting for the condition",
        }
        pod "pod-01efdf3c-ea61-4e77-bfe6-27ebd122ab34" is not Running: timed out waiting for the condition
    occurred

... skipping 59 lines ...
May 11 01:15:06.296: INFO: Waiting for pod external-injector to disappear
May 11 01:15:06.359: INFO: Pod external-injector still exists
May 11 01:15:08.297: INFO: Waiting for pod external-injector to disappear
May 11 01:15:08.359: INFO: Pod external-injector still exists
May 11 01:15:10.296: INFO: Waiting for pod external-injector to disappear
May 11 01:15:10.357: INFO: Pod external-injector no longer exists
May 11 01:15:10.357: FAIL: Failed to create injector pod: timed out waiting for the condition

Full Stack Trace
k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumesTestSuite).DefineTests.func3()
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186 +0x485
github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes.TestEBSCSI(0x4082b9?)
	/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/tests/e2e-kubernetes/e2e_test.go:82 +0x311
... skipping 17 lines ...
May 11 01:15:25.925: INFO: At 2022-05-11 01:09:59 +0000 UTC - event for ebs.csi.aws.com4rfmk: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
May 11 01:15:25.925: INFO: At 2022-05-11 01:09:59 +0000 UTC - event for ebs.csi.aws.com4rfmk: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } Provisioning: External provisioner is provisioning volume for claim "volume-184/ebs.csi.aws.com4rfmk"
May 11 01:15:25.926: INFO: At 2022-05-11 01:10:00 +0000 UTC - event for ebs.csi.aws.com4rfmk: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
May 11 01:15:25.926: INFO: At 2022-05-11 01:10:03 +0000 UTC - event for ebs.csi.aws.com4rfmk: {ebs.csi.aws.com_ebs-csi-controller-6dfc5c4d64-2lrrf_bd4098e5-3365-4ef7-a873-3dcd5bea6ec3 } ProvisioningSucceeded: Successfully provisioned volume pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2
May 11 01:15:25.926: INFO: At 2022-05-11 01:10:04 +0000 UTC - event for external-injector: {default-scheduler } Scheduled: Successfully assigned volume-184/external-injector to ip-172-20-58-208.us-west-2.compute.internal
May 11 01:15:25.926: INFO: At 2022-05-11 01:10:07 +0000 UTC - event for external-injector: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2" 
May 11 01:15:25.926: INFO: At 2022-05-11 01:10:19 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMapVolume: MapVolume.MapPodDevice failed for volume "pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2" : rpc error: code = Internal desc = Could not mount "/dev/nvme1n1" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2/6bca25ca-be09-45e2-8996-e9f65094fa70": mount failed: exec: "mount": executable file not found in $PATH
Mounting command: mount
Mounting arguments:  -o bind /dev/nvme1n1 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-87f56ce4-9d4d-443b-a3e5-f938d3eacbe2/6bca25ca-be09-45e2-8996-e9f65094fa70
Output: 
May 11 01:15:25.926: INFO: At 2022-05-11 01:12:06 +0000 UTC - event for external-injector: {kubelet ip-172-20-58-208.us-west-2.compute.internal} FailedMount: Unable to attach or mount volumes: unmounted volumes=[external-volume-0], unattached volumes=[default-token-f77jn external-volume-0]: timed out waiting for the condition
May 11 01:15:25.987: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 11 01:15:25.987: INFO: 
... skipping 108 lines ...
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (block volmode)] volumes
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should store data [It]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:159

    May 11 01:15:10.357: Failed to create injector pod: timed out waiting for the condition

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/volumes.go:186
------------------------------
S
------------------------------
[BeforeEach] [Testpattern: Pre-provisioned PV (default fs)] subPath
... skipping 47 lines ...

S [SKIPPING] in Spec Setup (BeforeEach) [0.001 seconds]
External Storage [Driver: ebs.csi.aws.com]
/home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/external/external.go:175
  [Testpattern: Dynamic PV (ntfs)][Feature:Windows] subPath
  /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:50
    should fail if subpath directory is outside the volume [Slow][LinuxOnly] [BeforeEach]
    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/testsuites/subpath.go:240

    Driver ebs.csi.aws.com doesn't support ntfs -- skipping

    /home/prow/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/vendor/k8s.io/kubernetes/test/e2e/storage/framework/testsuite.go:121
------------------------------
SS{"component":"entrypoint","file":"k8s.io/test-infra/prow/entrypoint/run.go:169","func":"k8s.io/test-infra/prow/entrypoint.Options.ExecuteProcess","level":"error","msg":"Entrypoint received interrupt: terminated","severity":"error","time":"2022-05-11T01:16:05Z"}
++ early_exit_handler
++ '[' -n 171 ']'
++ kill -TERM 171
++ cleanup_dind
++ [[ true == \t\r\u\e ]]
++ echo 'Cleaning up after docker'
... skipping 5 lines ...